Ongoing aggravation with data being harvested from web sites.

[Spacetrucker]

I'm running R60, soon to be upgrading to R65. What I need is "If x number of packets are received in x number of minutes from the same source ip address then drop all packets from that address for x number of minutes." Can this be done? Can it be done without writing a custom script? Can it be done by writing a custom script? I've tried Smart Defense -> Network Security -> IP and ICMP -> Network Quota. I set it to as little as 5 connections per second from the same source then drop all packets from that source for 5 minutes. It didn't stop the harvesting from the source ip address I needed to block, but it did stop some other legitimate traffic. And, also the SmartDefense -> Port Scan -> Host Port Scan and Sweep Scan. What good does that do anybody to just detect it? Why not be able to config that setting to drop packets from those source ip addresses? Same thing for the SmartDefense -> DoS settings, if they're detected, allow me to config it so it drops the packets. These seem like such simple requests that it already would and if not, then it should be built into the product. This is a great forum and I've learned a lot from the members and I appreciate that transfer of knowledge. But, I'm thinking it's time to look at some other firewall products out there. This isn't getting it done for us.

[Doeschi]

hmm, I don't think this is the main job of a stateful inspection firewall. More it
sounds like a task for either a real IPS/IDS (not SmartDefense) or some kind of
tuning the web server configuration.

[Spacetrucker]

I agree it's not the main job. But as long as CP is going to throw SmartDefense into the mix give it some real bite and remove the bark. Or take it out of the mix all together and stop misleading people with the hype. I've seen a line or two in this forum along the lines of "If it don't make sense, then it must be SmartDefense" or "Maybe CP will get it right the next time". CP don't give jack away, even their Knowledge Base articles have a cost, and for what we pay for the product, it should be right this time, not the next. And if CP can provide a SD solution for too many concurrent connections from the same source, that drops the packets from that source, it shouldn't be that far of a stretch to provide it for the others. Umm, can you provide answers to the questions I've asked?

[Thorpuse]

Eventia Analyser will do this with the event triggers it provides. That's your easiest path.

[rubber_chicken]

Dumb question, but from reading this thread, you're trying to stop some malicious IP from scraping the entire content of your <public> website(s)? Please correct me if I'm wrong.

If I'm not, then fundamentally you're doing the wrong thing of even posting data on the web? Do you need to go back to the drawing board and look at the security model you are working with? (securing the website/using a suitable model of authenication/classifying data correctly for public consumption and so on)

If the data is public, then it is public -you don't control it. Trying to block an IP isn't going to help you.

[Spacetrucker]

Thorpuse - thanks for the suggestion. I'll follow up on it. Rubber Chicken - You do understand the problem. But just because data is meant to be publicly available, doesn't give certain members of the public the right to scalp the data from the sites using routines that prevent other public users from accessing the data. If they need the data in bulk, they can contact us and we'll send it to them. I'm not trying to control the data, I'm trying to control the rate at which it's being accessed at the firewall. The developers know they can control that access rate at the web server by writing some code. But if I can cut the offending source off at the firewall. Less bandwidth is consumed internally and you reduce the load on the webservers. That increases performance along with the response times.

>>If the data is public, then it is public -you don't control it. Trying to block an IP isn't going to help you.

You couldn't be more wrong.

[BarryStiefel]

Originally Posted by Spacetrucker

Thorpuse - thanks for the suggestion. I'll follow up on it. Rubber Chicken - You do understand the problem. But just because data is meant to be publicly available, doesn't give certain members of the public the right to scalp the data from the sites using routines that prevent other public users from accessing the data. If they need the data in bulk, they can contact us and we'll send it to them. I'm not trying to control the data, I'm trying to control the rate at which it's being accessed at the firewall. The developers know they can control that access rate at the web server by writing some code. But if I can cut the offending source off at the firewall. Less bandwidth is consumed internally and you reduce the load on the webservers. That increases performance along with the response times.

>>If the data is public, then it is public -you don't control it. Trying to block an IP isn't going to help you.

You couldn't be more wrong.

Is it possible the scrapers will read and respect a robots.txt file?

[Spacetrucker]

Barry, thanks for the suggestion, but they won't. We do have a robots.txt file in place. I'm following up on Thropuse's suggestion on Eventia Analyzer.

[Spacetrucker]

Barry you ever work in the Netware forums?

[BarryStiefel]

Originally Posted by Spacetrucker

Barry you ever work in the Netware forums?

No, I was never a Netware guy.