detect/address: validate netmasks

Only accept netmask in dotted quad notation if they can be turned
into a CIDR.

According to rfc 4632, CIDR (compat) netmasks are all that should be
used.

Bug: #5168.

src/detect-engine-address.c

@@ -456,6 +456,16 @@ static int DetectAddressParseString(DetectAddress *dd, const char *str)
                     goto error;
 
                 netmask = in.s_addr;
+
+                /* validate netmask */
+                int cidr = CIDRFromMask(netmask);
+                if (cidr < 0) {
+                    SCLogError(SC_ERR_INVALID_SIGNATURE,
+                            "netmask \"%s\" is not usable. Only netmasks that are compatible with "
+                            "CIDR notation are supported. See #5168.",
+                            mask);
+                    goto error;
+                }
             }
 
             r = inet_pton(AF_INET, ip, &in);