Blog

    How to Save a Million Dollars on Your IT Costs

    Inexperienced IT engineers tend to guess at causes of network problems.  I see this time and time again and it drives me crazy.  They guess and then discuss their guesses until they decide which guess is the best.  I’ve watched ‘engineers' sit together in the IT office and seriously discuss their random guesses for half an hour or more before taking action on what might, or might not, be the actual issue.  

    If you are a CTO or an IT manager and are thinking “isn’t that what engineers are supposed to do?” then hang on… this post is for you.

    Bona fide experienced technical engineers will approach the same problem by reviewing the system configuration, searching any available log files for errors and then will run ‘debugging’ commands on the equipment.  Debugging means that the engineer sets the software on the equipment to emit a verbose audit trail of the system’s actions into a text file (A.K.A log file or syslog).  The engineer can then search the log files to see what the system is ‘saying’ about the problematic service or process.

    The rather obvious difference in the two approaches is that experienced engineers search for facts first and then form a view as to why the service is broken or misbehaving.  They don’t bother with guessing before they have evidence because that wastes time, resources and money.

    So take this simple difference in approach and look at it in the context of a global project; a new call centre for instance.

    I’ve sat through hundreds of all night conference calls with engineers from London or New York to Bangkok or Mumbai where the engineers have no access to view the system log files for the services from end to end.  The calls go on and on as one engineer sends clips of log files to the group for one device at a time - usually via unencrypted e-mail.  The process is slow and expensive.  The engineers have so little information that they still end up guessing at what is wrong.  Hours go by and eventually more calls are scheduled when other specialists can join to continue troubleshooting (or, more accurately, to continue guessing). 

    These conference calls can cost the organisation over $10k per night (10 contracted engineers and two project managers all working double shifts).  One hundred calls later you’ve spent a million dollars.  If that sounds unrealistic consider a project with four work streams; that’s only 25 conference calls per work stream.  The painful truth is that many of these calls fail to progress the project because the log data from the systems that make up the service is not shared properly with the engineers.

    So how do you ensure that this doesn’t happen to you?  Before you start up a project consider the following:

    • Employ a collaborative troubleshooting platform
      • These platforms exist; I like S4NITY (www.s4nity.com) because I helped to build it but there are others
      • Agree with internal teams and relevant vendors that, at least for the duration of the project, ALL of the log files from ALL of the equipment that enables the service will be sent to the one collaborative troubleshooting platform
      • Ensure that the platform is capable of indexing the log files for fast search 
      • Ensure that project engineers proactively review the log file data via the platform daily and use this platform as the starting point to troubleshoot issues as and when they arise
      • Ensure that the search portal is the centre piece of troubleshooting sessions so that everyone has access to the information

    OK - so what about business as usual?

    There are two reasons, in my mind, that every business needs to proactively employ a collaborative troubleshooting platform for BAU:

    • You may have to make do with fewer resources and higher workloads per person
      • The unemployment rate is falling sharply (in London, source: data.london.gov.uk)
        • There are not enough skilled engineers available to satisfy demand
    • More handheld devices, sensors and other equipment with IP interfaces (think Internet of Things) are joining the corporate network
      • The more you can proactively collect, index, aggregate, search and report on the log files generated by this equipment the better chance you will have of supporting the organisation

     

    Just to stay above water your team will need the right tools to work efficiently and cut time to fix when issues occur.

    So, in summary, remember that troubleshooting (either on projects or BAU) should involve gathering the evidence first and then making a call based on the evidence, not guessing, and that if you setup a system to proactively gather evidence and make it available to the engineers that can use it you have a better chance at cutting down on time to fix and maybe avoiding some problems altogether.   This will save you a lot of money. 

     

    Blog tags

    How to Configure External Logging on an SRX Secure Gateway

    Please view the related screen cast here.

    There are two types of logs that you can send to a s4nity.com logging server from the SRX Secure Gateway.  

    1. The 'security' logs contain information about what is traversing the gateway; these can be used to understand and troubleshoot problems with traffic flows
    2. The 'syslog' logs contain information about the gateway itself; these will be used to gather information about who has logged into the device and what changes have been made

     

    To configure either type of logging you must know the IP address of your logging server and the source address (on the SRX) from which you will send the log traffic.  The source address is normally the interface that is closest to your logging server.  For this example I will be using the log server address '172.16.25.10' and the SRX source interface IP address will be '172.16.25.1'.

     

    The steps to configure the security logs are as follows: 

    1. set security log mode stream

      • This sets the mode to streaming

    2. set security log stream s4nity severity debug

      • This tells the SRX how much information to log - here we are logging everything by specifying debug

    3. set security log stream s4nity format sd-syslog

      • This tells the SRX what mode to send the logs in; sd-syslog sends the logs in nice key value pairs

    4. set security log stream s4nity category all

      • This tells the SRX which category of traffic logs to send to the logging server; we want all 

    5. set security log stream s4nity host 172.16.25.10

      • This tells the SRX which log server to send logs to; here we are using a s4nity local collector or syslog server with the IP address of 172.16.25.10

    6. set security log stream s4nity host port 5543

      • This tells the SRX which port to send the logs over - this port must be configured on the s4nity local collector to listen for logs

    7. set security log source-address 172.16.25.1

      • This tells the SRX which source IP to send the log from

     

    The steps to configure the sylog are as follows: 

    1. set system syslog host 172.16.25.10 any any

      • This tells the SRX the IP of the s4nity local collector or syslog server and which logs to send to it; in this case we are sending all logs

    2. set system syslog host 172.16.25.10 structured-data

      • This tells the SRX which format we want to send the logs in; we are using structured-data which (nice key value pairs)

    3. set system syslog host 172.16.25.10 port 5542

      • This tells the SRX which port our s4nity local collector is listening on for these system logs

    4. set system syslog host 172.16.25.10 source-address 172.16.25.1

      • This tells the SRX which source interface we want to send the logs from

    Automation will help your team with IT security, documentation, compliance and performance.

    I am already looking forward to 2015 and with the challenges and benefits of compliance, security and infrastructure performance management there is one thing that I am really excited about.  Automation.  Hmmm, really?  Yep!  Automation needs to be at the top of the list of technology to embrace in 2015.  

    So, what does Automation mean?  To me it means using tools to create, document and implement a workflow to manage and deploy all of your organisation's software configurations.  This means everything from your website/services to your networking equipment, telephone switches, desktops and tablets.  Why would you do this?It makes you think before you make a change.  The idea is that you setup templates that contain the current best configuration for a type of system and that you can update and push that template out to any or all systems of that type.  Sounds like group policy right?  It is system independent.  You can manage all of your systems from the same core set of templates.  This encourages planning and change control because you need to change the template prior to pushing it to any live systems.  Which means you need to think it all out beforehand.  Also, with a good system templates are versioned.  Rolling back to the previous working version can be done quickly if a problem appears with a new configuration. 

    Automation also improves your documentation; again, this is because the templates serve as your documentation.  Every current good configuration will be in the system.  When teams want to make changes they can check out the current configuration template, update it, check it in and push it when appropriate.  As a result your security posture improves.  You introduce continuity across systems, you know your inventory and well documented systems help teams to think more strategically and to consider impact of changes.  This can only help your compliance initiatives. 
     
    So how does Automation differ from deployment scripts?  Most organisations have lots of scripts used for IT operations.  Some common problems with scripts include:
    • they are written in several different scripting languages,
    • they are local to a team or just one person,
    • they are not in version control and are not easily deployed or rolled back.
     
    The Automation systems available today (Chef, Puppet, CFEngine) all address the common deployment script problems.  If your team uses a set of scripts to manage your deployments it is likely that only your team knows how to use them.  If you use Chef there are thousands of teams using the same tool set with similar templates.  Thousands of people are sharing their experience in forums.  You can hire a new admin that knows how to use the Automation tool or ask your admins to read a book on how to use the given tool set.  
    How do we get started?  Buy a book.  Seriously, go to Amazon.com and search for Automation, Chef, Puppet or CFEngine.  Search YouTube for the same - there are plenty of educational clips on automation.  You will probably see some presentations given by the folks who make the tools.
    How do you make a significant and lasting change to your organisation with Automation?  Find someone to help you implement it.  Not to do it for you but to assist you on an ongoing basis to manage the tools.  Your team will pick up the Automation if you give them a chance but don’t throw yet another system on their plate (to be managed by them) or it will never get off the ground.  S4nity.com can help to deploy and manage the tool set.  This is what we do.  There are others though - so look around, talk you people, gather information, make a call on what tool to use and then jump in.  
    2015 will be a great year to introduce Automation; the time and money you spend in the beginning will be well worth it.  Your team will learn new skills and your organisation will benefit from a well documented, secure and compliant system.
     
    Seth Curry
    s4nity.com limited
    November 2014

    Juniper SRX Service Gateway Branch Office Configuration Example

    This cheat sheet will help you to get started with configuring a Juniper SRX Branch office firewall.
    The enclosed commands should work on version 12.1 and above.

    Set some system items up to get started

    Set the root password

    #set system root-authentication plain-text-password

    Set the host-name

    #set system host-name MYSRXFIREWALL

    Set the name server that the firewall will use

    #set system name-server 8.8.8.8

    Set a new username to admin the firewall with

    #set system login user adminuser class super-user

    #set system login user adminuser authentication plain-text

    Setup some zones, we will use RED for untrusted, GREEN for trusted and YELLOW for DMZ

    #set security zones security-zone RED description "ch: initial install desc: Untrusted Internet"

    #set security zones security-zone YELLOW description "ch: initial install desc: DMZ"

    #set security zones security-zone GREEN description "ch: initial install desc: trusted LAN"

    Set the interfaces and put them in their new zones

    #set interfaces fe-0/0/0 unit 0 family inet address 10.12.0.4/24

    #set security zones security-zone RED interfaces fe-0/0/0.0

    #set interfaces fe-0/0/1 unit 0 family inet address 192.168.25.0.1/24

    #set security zones security-zone GREEN interfaces fe-0/0/1.0 host-inbound-traffic system-services ssh

    #set security zones security-zone GREEN interfaces fe-0/0/1.0 host-inbound-traffic system-services dhcp

    #set security zones security-zone YELLOW interfaces fe-0/0/2.0 family inet address 172.16.25.0.1/24

    ** Note that we have allowed the ssh and dhcp services inbound to the GREEN zone interface fe-0/0/1.0

    **this is so that we can manage the firewall from the GREEN zone using ssh and also we will setup a DHCP server for the GREEN zone

    Set the static route to be out to the Internet via the RED zone

    #set routing-options static route 0.0.0.0/0 next-hop fe-0/0/0.0

    Set an address book entry in the global address book for the GREEN zone network

    #set security address-book global address greenNetwork 192.168.25.0/24

    Setup a rule to allow our trusted GREEN zone traffic out to the Internet via out untrusted RED zone over http

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP description "change: initial install desc: Internet outbound"

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP match source-address greenNetwork

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP match destination-address any

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP match application junos-http

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP then permit

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP then log session-init

    #set security policies from-zone GREEN to-zone RED policy GREEN_TO_RED_HTTP then log session-close

    Setup a new application and put it into an application-set

    #set applications application imaps protocol tcp

    #set applications application imaps destination-port 993

    #set applications application-set EMAIL_SERVICES application imaps

    #set applications application-set EMAIL_SERVICES application junos-mail

    #set applications application-set EMAIL_SERVICES application junos-pop3

    Setup an address-book for the YELLOW zone and attach it to the zone

    #set security address-book YELLOWBOOK address mailServer description "change: initial install desc: e-mail server in DMZ"

    #set security address-book YELLOWBOOK address mailServer 172.16.25.200/32

    #set security address-book YELLOWBOOK address-set EMAIL_SERVERS description "change: initial install desc: e-mail server group"

    #set security address-book YELLOWBOOK address-set EMAIL_SERVERS address mailServer

    #set security address-book YELLOWBOOK attach zone YELLOW

    Setup a policy to allow any Internet address (from-zone RED) to the e-mail server (to-zone YELLOW) using the application-set that we created earlier

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL description "change: initial install desc: Internet inbound rules for e-mail server access"

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL match source-address any

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL match destination-address EMAIL_SERVERS

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL match application EMAIL_SERVICES

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL then permit

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL then log session-init

    #set security policies from-zone RED to-zone YELLOW policy RED_TO_YELLOW_EMAIL then log session-close

    Setup dhcp on the GREEN zone; this is the new way to setup dhcp and can be used within routing instances

    #set system services dhcp-local-server group JDHCP_GREEN interface fe-0/0/1.0

    #set access address-assignment pool JDHCP_GREEN_POOL family inet network 192.168.25.0/24

    #set access address-assignment pool JDHCP_GREEN_POOL family inet range JDHCP_GREEN_RANGE low 192.168.25.5

    #set access address-assignment pool JDHCP_GREEN_POOL family inet range JDHCP_GREEN_RANGE high 192.168.25.30

    #set access address-assignment pool JDHCP_GREEN_POOL family inet dhcp-attributes name-server 8.8.8.8

    #set access address-assignment pool JDHCP_GREEN_POOL family inet dhcp-attributes router 192.168.25.1

    SRX Set Time Zone and DNS Server

    When you get started with your firewall configuration it is really important to add the time server settings. This will ensure that when you are looking at log files the time is accurate.
    I start by setting the name server so that the firewall can resolve DNS names. This allows me to use the NTP server pools by name and increases the chances that I will get a working time server.

    First enter configuration mode:

    #configure

    Now we can set the DNS name server to use; in this case I will use the google name server:

    #set system name-server 8.8.8.8

    Now we can set the time zone:

    #set system time-zone Europe/London

    We can manually set the date to start by telling the firewall to get the date from the NTP server:

    #set date ntp 0.pool.ntp.org

    And finally we can set the time servers:

    # system ntp server 0.pool.ntp.org version 4 prefer
    #set system ntp server 1.pool.ntp.org version 4

    Remember that on Juniper you need to commit the config:

    #commit and-quit

    To check the time server status and association I will use the following commands:

    >show ntp status

    >show ntp association

    Juniper SRX Service Gateway FreeRadius Example

    This blogs shows an example of a working FreeRadius configuration for the Juniper SRX 210H Service Gateway.
    Note: mschapv2 can be set on the SRX but does not work with Freeradius; however, passwords are sent as MD5 hashes

    In the following configuration we are using the trust interface as a source but will be changing this to use the loopback - this will require rules to allow udp 1812-1813 from management to trust

    set version 11.4R10.3

    set system authentication-order radius
    set system authentication-order password

    set system radius-server 192.168.10.25 port 1812
    set system radius-server 192.168.10.25 accounting-port 1813
    set system radius-server 192.168.10.25 secret “******”
    set system radius-server 192.168.10.25 source-address 192.168.1.1
    set system radius-options attributes nas-ip-address 192.168.1.1

    #Accounting commands
    set system accounting events login
    set system accounting events change-log
    set system accounting events interactive-commands
    set system accounting destination radius server 192.168.10.25 port 1812
    set system accounting destination radius server 192.168.10.25 accounting-port 1813
    set system accounting destination radius server 192.168.10.25 secret “*****”
    set system accounting destination radius server 192.168.10.25 source-address 192.168.1.1

    set system login user remote full-name "default remote access user template"
    set system login user remote uid 2003
    set system login user remote class read-only
    set system login user super-users uid 2002
    set system login user super-users class super-user

    Freeradius config file: /etc/raddb/users

    NOTE: There is also some cisco config; this allows the cisco ASA and SRX to reference the same user accounts
    NOTE: radcrypt (part of the radius tools package) was used to create the encrypted passwords - not scaleable without some script to help users
    to change passwords and encrypt them every x number of days

    username1 Crypt-Password := “******”
    Service-Type = Administrative-User,
    Cisco-AVPair = "shell:roles=network-operator",
    Cisco-AVPair += "shell:priv-lvl=1",
    CVPN3000-Privilege-Level = 5,
    Service-Type = Login-User,
    Juniper-Local-User-Name := "remote"

    username2 Crypt-Password := “*****”
    Reply-Message = "Hello, %{User-Name}",
    Service-Type = Administrative-User,
    Cisco-AVPair = "shell:roles=network-admin",
    Cisco-AVPair += "shell:priv-lvl=15",
    CVPN3000-Privilege-Level = 15,
    Juniper-Local-User-Name := "super-users"

    Freeradius config file: /etc/raddb/client.config

    ient 192.168.1.1 {
    secret = xxxxxxx
    shortname = 1st-lon-srx
    nas_type = juniper
    }