Networking

Networking

There are tons of simple mistakes, oversights, omissions, and bad ideas that can bite you in the networking realm. Here are some of our favourites. Whether they caused outages, security issues, poor performance, or just made troubleshooting a pain, we’ve been there, done that, bought the t-shirt.

DNS

  1. Forgetting to close a zone name with a trailing dot. The lack of one simple period stopped mail for 30K users once at a major company.
  2. Leaving your zone open to perform a full zone transfer to any requestor just makes recon easy for attackers. Never permit zone transfers to any server except the ones you want hosting a copy.
  3. Setting week long TTLs may reduce traffic, but when you make a typo and don’t realize it for a couple of hours, explaining that it will take a week before that site is back up globally, is an uncomfortable conversation to have.
  4. Keeping all your DNS servers on the same network is a quick path to losing everything when a routing table entry goes bad.
  5. Using routing/NAT instead of split DNS is not the way to let your users access DMZ resources using external names; that is, not if you expect everything to work.
  6. Circular recursion is when two different servers are configured to recurse to one another for unknown requests. You can take out an entire Ethernet segment if the DNS servers are robust enough as the request ping-pongs between them.
  7. It may seem like a pain to create all the in-addr.arpa records so you can use PTR records internally, but when you are trying to find the source of some rogue traffic, you’ll be glad you did.

Other networking issues

  1. Yes, I know Microsoft told you 10 years ago that NetBIOS was going away. Guess what. It’s still here. Not configuring WINS is a bad idea and can cause poor network performance.
  2. Clear-text protocols should never be used, no matter how simple they are to set up or how limited you think their use will be. Ignore this rule and soon you will see domain creds crossing the wire.
  3. Using non-standard ports is just a bad idea all around. You may think it lets you do more on your server, or provides some security, but all it really does is keep your customers from accessing the services. Way too many potential customers could use your web app if it was on 80 or 443, but their network blocks outbound access to anything else.
  4. Legacy rules on the firewall must be cleaned up immediately. Open ports to unused ip.addrs are a disaster waiting to happen when a new box uses an old address.
  5. NAT internal traffic out to the Internet, and Internet traffic to your DMZ for applications that support it, but never NAT internally. You will break stuff left and right, and some of that cannot be fixed. There once was a major corporation who ignored that, and wound up having to migrate 10,000 user accounts, twice, because they couldn’t get a trust working through a NAT’d connection.
  6. Using long xlate timeouts means that bad connections stick around for hours. Think of xlates in terms of seconds to minutes, but never in hours.
  7. Bypassing the firewall is never the right answer. If you are going to allow systems to span the external and internal network, and they themselves are not firewalls, you may as well just get rid of the firewall entirely. Think of the money you’ll save on annual support. You’ll need that to pay for incident response!
  8. Never use a domain name that you do not own. Register anything you want to use, even if you think it will never be used externally.
  9. Never use bogon addresses. They won’t be bogons forever. RFC 1918 addresses and registered addresses you control are the only ones you should use.
  10. Using a HOSTS file to get around something, and then never going back to fix the issue. It’s never the HOSTS file, until you spend a few hours troubleshooting DNS before you finally remember that it actually is the HOSTS file.

User accounts and passwords

  1. Go Bing Search default password lists, and then go change all the default passwords you never thought were a risk.
  2. Never set a user’s password to never expire. We expire passwords so that when they are compromised, we limit the time the attacker can use the compromised credentials. Never is a mighty long time.
  3. Don’t forget to set a service account’s password to not expire, and then schedule a regular reset. If a service account’s password expires, the next time that service needs to start, it won’t.
  4. Setting too complex a password requirement just leads users to write things down.
  5. Requiring too many things in password (must start with a capital, must end in number, must be no more than 11 characters long, etc.) leads to passwords that will be written down.
  6. Using the same password for multiple accounts means that an attacker has the master key as soon as he compromises one service. Use different passwords for your work, personal email, Facebook account, etc.
  7. Never use a User account as a service account. You change your password, you break the service.
  8. Leaving sAMAccountName blank may be fun and make scripting easier, but there are some systems that cannot use UPN.
  9. Using non-routable UPNs works great internally, but the more services that move to the cloud, the more important a routable UPN will be.
  10. Never set the local admin account password to be the same on all workstations. Someday, it will get out, and you’ll have to change it on every single machine in the company. Don’t use a pattern that can be easily figured out either.
  11. Stale accounts should be disabled and then deleted.
  12. Weak passwords are worse than strong ones written down. An attacker has to find the piece of paper you wrote a password on; they can break a weak password from anywhere.
  13. If you have a web service you offer to external users, like your customers, let them use their email address as their username. It’s unique, you don’t have to maintain it, and they will be much happier for it.

Speaking of Email…

  1. As much as they are spam magnets, you still need to establish the RFC 2142 addresses and monitor them. Comply with the RFCs.
  2. The last thing you should do to a mail server before opening port 25 on the firewall is to check it for open relay.
  3. The first thing you should do to a mail server after opening port 25 on the firewall is to check it for open relay.
  4. Emailing a bunch of people, some of whom don’t work for you? Use BCC. CC’ing hundreds of strangers happens by accident all the time. That they immediately drop your service and block your emails after this happens is not by accident.
  5. Spellchecking is not optional! Enable it on your client, and check the spelling of your email, and your subject line!
  6. Reply to all is almost never a good idea. Consider how much noise you deal with every day in your inbox, and do unto others as you wish they would do unto you.

Active Directory

  1. Never set up a DC to suffer DNS islanding. A DC should never use itself for DNS; always point it at another DC.
  2. Keep your admin accounts separate and use a second, normal account for your workstation use. Too many things like inheritance filtering can cause administrative accounts all kinds of problems with other services.
  3. Monkeying with inheritance is a bad idea. I’ve never seen a case where it was a good idea, but I’ve seen literally hundreds where someone turned off inheritance years ago, and it caused major issues for the poor team that ‘inherited’ the mess.
  4. Repeat after me. You create a new subnet, you define it in AD and assign it to a site. It takes two extra minutes. Not keeping up with your sites will cause logon and other performance issues, can impact replication, and will be a multi-month project to clean up when it finally becomes the blocker to a major project.

Antivirus

  1. Disabling antivirus should be a capital offense.
  2. Not configuring proper exemptions based on vendor recommendations is a sure way to crush application performance, and will cause others to disable antivirus protection.
  3. Not investigating failed updates immediately is asking for trouble, since usually the reason an update failed is either because the antivirus was disabled, or your machine is already infected.

Teach your Users well

  1. Asking for their password is never acceptable. The first time you do, you can never again expect them to safeguard their credentials from potential attackers.
  2. Using self-signed certs just trains your users to ignore certificate warnings and makes them click-through-happy. The last thing you want users to do is ignore warnings.

Other

  1. Certificates expire. Not monitoring certificates will lead to outages, and mad scrambles to replace them. Set up a reminder for a couple of weeks ahead of expiration so you have plenty of time to renew.
  2. Pings are not the enemy. Not allowing ICMP echo just makes troubleshooting and simple monitoring more difficult, and it violates RFC 792 and 4443. Don’t violate the RFCs.
  3. Not using NTP is just lazy. Pick a good time source and sync everything to it. It makes your logging correlation much easier, makes users happy, and prevents authentication from breaking.
  4. Not verifying backups is playing Russian Roulette. You must perform regular test restores to ensure you have a good backup.
  5. Deploying an untested patch is only  fractionally worse than…
  6. Skipping a patch.
  7. Encrypt everything that can leave the four walls of your network. That means laptop hard drives, USB keys, and network traffic. Unless it’s data you would post on your website, it should never go out the door (physical or virtual), unless it’s encrypted.
  8. Using unsupported configurations may be cheaper or faster, but unless you really are willing to forgo the option of vendor support, it’s just not worth it. You’ll spend more money trying to troubleshoot an issue, or waste more time getting things back into a supported configuration than you ever saved by trying to cut corners.
  9. I have personally seen at least three instances where letting support/maintenance lag or expire wound up costing the companies in question millions of dollars. Never let your software support expire until you have formatted every machine that was running that software.
  10. Staying on legacy software versions may save you upgrade fees, but eventually you will find yourself in an unsupported configuration, or at a point where you want to do X, but you cannot because it won’t support legacy Y. You’ve just found that proverbial spot between the rock and the hard place.
  11. Hardcoding anything – just don’t do it.
  12. Documentation is a pain. We all hate it, we never do it, and yet we curse the guy who came before us that didn’t keep good documentation. Pay it forward.

 

System Admin Oath

In order to defend and preserve the honor of the profession of systems engineers and administrators, I Promise that, to the best of my ability and judgment:

1. I will not create harmful infrastructure or systems.

2. The infrastructure and systems that I create will always be my best work. I will not knowingly make changes to infrastructure or systems that are defective either in behavior or structure.

3. I will produce, with each change, a quick, sure, and repeatable proof that every element of the infrastructure and systems work as it should.

4. I will make frequent, small, changes so that I do not impede the progress of others.

5. I will fearlessly and relentlessly improve the infrastructure and systems at every opportunity. I will never make the infrastructure worse.

6. I will do all that I can to keep the productivity of myself, and others, as high as possible. I will do nothing that decreases that productivity.

7. I will continuously ensure that others can cover for me, and that I can cover for them.

8. I will produce estimates that are honest both in magnitude and precision. I will not make promises without certainty.

9. I will never stop learning and improving my craft.