Monday Morning notes – 03.02.2020

Greetings programs!

It’s been quite a ride for the last month or so, so much going on. Last Monday morning I passed the new Cisco DEVASC exam (200-901) on the first day to test, making me a member of the Devnet 500 club.

DEVNET 500 w00t!

Preparing for the exam really did a lot for sharpening up some relatively new skills I’ve acquired that haven’t had a lot of reinforcement. That’s the real value of IT certifications – having that performance-based focal point. When your learning is self-directed there is a real tendency to sort of wander from topic to topic and lose focus, and learning tracks are a great tool to combat that.

That said, I think the most important takeaway is that we’re rapidly transitioning to an API driven world and it’s important to get comfortable working with structured data and logical abstractions in general regardless of the job role.

Ok, about Terraform. What I’m working through is the approach I want to take. I want to bring a unique angle rather than join a chorus of people all saying and doing the same thing. If there are specific things you would like to see, let me know in the comments.

Thanks for reading and I’ll see you around.

Monday morning note 02.17.2020

Greetings Programs!

Just a quick check-in for this week.

Life has gotten a bit hectic recently, and will likely remain so for another couple of weeks. Small things are beginning to take conscious effort and focus which is a reliable indicator that I need to trim the sails a little. What this means for this space is the first Terraform post is delayed for a week, possibly two.

On a brighter note, the weather was wonderful for this morning’s walk, my dogs were well behaved, and coffee is always there for us and wants us to be happy.

Until next time, may you care for yourself with ease.


PKIFNE #12: Diffie Hellman Groups

Greetings programs!

Today’s post is an excerpt from my CCIE Security notes, and it has a useful table of Diffie Hellman groups, which you can use as a job aid in IPSEC VPN designs. I Hope you find it useful.

About Diffie Hellman

The key bit of magic that makes IKE (Internet Key Exchange) possible is Diffie-Hellman. Diffie-Hellman allows anonymous entities to calculate a shared secret that can’t be discovered by a third party listening to the exchange. What’s amazing about it is the peers are able to do this using two different passwords that they keep private and never exchange. DH is one of the earliest examples of Public Key Cryptography.

Key points

  • Diffie-Hellman does not provide authentication. It’s used to assist in creating a secure channel for authentication
  • Diffie-Hellman does not provide encryption. It provides the keying material for encryption.
  • Diffie-Hellman is used for control plane functions only.


At a high level it works like this:

If side a and side b use the same generator and modulus, resulting values from step 5 and 6 should be the same. This shared key is used as an input to the negotiated encryption algorithm.

In the case of Diffie-Hellman The generator and Prime (g,p) are predefined values (defined in a number of different RFCs) which are referenced as Diffie-hellman groups. The larger the Generator and Prime are, the more difficult it is to break. As computational power has increased substantially since the first DH groups were defined, the old groups are no longer safe to use.

The following Table lists the Diffie-Hellman Groups:

Diffie Hellman Groups

*NGE refers to Cisco Next Generation Encryption, which is the vendors set of recommended ciphersuites.

*NSA Suite B refers to the United States the National Security Agency’s published list of list of interoperable modern cryptographic standards.

Children of the Magenta Line

Greetings programs!

I believe it’s a safe assumption that the current crop of budding network engineers are being indoctrinated with the notion that if they don’t 100% automate all the things, their jobs and careers will be at risk. My friend Daniel Dibb deserves ample credit for repeatedly asking the question “What happens when we subsequently end up with a generation of automation engineers who don’t understand networking?”.

Yesterday morning I was drinking coffee, scrolling through my Twitter feed, when I came across a thread where Marko Milivojević posted a link to a youtube video of American Airlines Captain Warren Van Der Burgh, delivering a talk in 1997 called Children of the Magenta Line. The context is lessons learned from a rash of airline crashes caused by flight crews becoming too dependent on automation.

If you are an infrastructure engineer, or you manage infrastructure engineers, this video is worth 25 minutes of your time.

There are two quotes that stand out from this Video:

  • We are Pilots and Captains, not Automation Managers
  • You’ve got to pick the appropriate level of automation for the task at hand

In googling the name of the video, I came across this article which had a quote from William Langewiesche which also illustrates a fundamental problem with over reliance on automation:

“We appear to be locked into a cycle in which automation begets the erosion of skills or the lack of skills in the first place and this then begets more automation.”

Let’s learn from the Airline industry and not repeat their mistakes.

Flock of birds wheeling around a statue of the barefoot mailman

Have a good one, and I’ll see you around.


Quick hit: playing with Terraform & IOS-XE Day 0 automation

Greetings Programs!

Over the last week or so I’ve been working on an Infrastructure as code Azure Hybrid cloud deployment, based on the Azure Spoke and Hub reference topology.

The gateways connecting Azure to the On-Prem datacenter in my case are Cisco Cloud Services Router 1000v (CSR1kv), using day 0 automation to inject a configuration into the Azure Router.

Here’s a link to the Azure Deployment Guide for the CSR1kv, which has some really interesting tidbits. I’ll dive into those in future post.

To actually execute the infrastructure build, I’m using a tool called Terraform.

Terraform does an incredible job of abstracting away all the details you normally have to worry about when provisioning cloud provider infrastructure. It’s a joy to work with. The documentation and examples are easy to follow, and the error messages tell you exactly what you did wrong and where to look.

Terraform Description file example

If that wasn’t cool enough, the Terraform binary is actually built in to the Azure Cloud shell. The net of that is, once you have your deployment files built, you can work with them using just cloudshell, which is super convenient.

Azure Cloud Shell – Checking the Terraform Version

That’s it for today. Lot’s of neat stuff in this lab that I’m dying to talk about, can’t wait to share.

Best regards,


Blowing the dust off, redux

Greetings programs.

I took this photo just a few minutes ago on my Monday morning beach walk. I hope you find it relaxing.

Holy cow, been 8 months since my last dispatch.  Still studying and learning cool stuff, have not done a good job at stopping to document my learnings, going to make another attempt to rectify that.

Been an eventful year.  In addition to learning DevOps tools and techniques and studying cloud native software design, I’ve also changed jobs.  As of August I’ve been working at Cisco systems as a pre-sales Architect.  It’s a dream job and I absolutely love working with my customers. I learn as much from them as they learn from me.

So yeah, the new job thing; several months of getting the firehose and struggling to take everything in and find my stride.

I didn’t feel like that was enough new stuff, so I enrolled in WGU’s Cybersecurity Bachelor’s program starting in January.  I don’t **need** a degree, but I don’t think it would hurt to become a more well-rounded technologist.  I’ve historically used certifications for learning paths which is fine but they tend to be narrowly defined in scope which can lead to polarized thinking.

So here’s the plan for the blog:  I’m committing to posting something every Monday morning, even if it’s just a “hello, I’m drinking coffee and checking my twitter feed how about you”.  I have a sizable collection of study notes, so there’s no shortage of meaningful topics however. Hopefully I’ll dip into that more than the small talk  :).

Have a great week, talk to you next Monday morning.


PKI for Network Engineers Ep 10: Cisco IOS CA introduction

Greetings programs!

In the next few PKI for network engineers posts, I’m going to cover Cisco IOS CA. If you’re studying for the CCIE security lab or you’re operating a DMVPN or FlexVPN network, and you’d like to use Digital certificates for authentication, then this series could be very useful for you.


IOS-CA is the Certification Authority that is built into Cisco IOS. While not a full featured enterprise PKI, for the purposes of issuing certificates to routers and firewalls for authenticating VPN connections it’s a fine solution. It’s very easy to configure and supports a variety of deployment options.

Key points

  • Comes with Cisco IOS
  • Supports enrollment over SCEP (Simple Certificate Enrollment Protocol)
  • RSA based certificates only
  • Easy to configure
  • Network team maintains control over the CA
  • Solution can scale from tiny to very large networks

Deployment Options

IOS-CA has the flexibility to support a wide variety of designs and requirements. The main factors to consider are the size of the network, what kind of transport is involved, how the network is dispersed geographically, and the security needs of the organization. Let’s briefly touch on a few common scenarios to explore the options.

Single issuing Root CA

For a smaller network with a single datacenter or a active/passive datacenter design, a single issuing root may make the most sense. It’s the most basic configuration and it’s easy to administer.

This solution would be appropriate when the sole purpose of the CA is to issue certificates for the purpose of authenticating VPN tunnels for a smaller network of approximately a hundred routers or less. Depending on the precautions taken and the amount of instrumentation on the network, the blast radius of this CA compromise would be relatively small and you could spin up a new CA an enroll the routers to it fairly quickly.

The primary consideration for this option is CA placement. A good choice would be a virtual machine on a protected network with access control lists limiting who can attempt to enroll. The CA is relatively safe from being probed and scanned, and it’s easily backed up.

Fig 1. Single issuing root

Offline Root plus Issuing Subordinate CAs

For a network that has multiple datacenters and/or across multiple continents it would make more sense to create a Root CA, then place a subordinate Issuing CA each datacenter that contains VPN hubs. By Default IOS trusts a subordinate CA, meaning the root CA’s certificate and CRL need not be made available to the endpoints to prevent chaining failure.

As in the case with any online/Issuing CA, steps should be taken to use access controls to limit access to HTTP/SCEP

Fig 2. Offline Root

Issuing Root with Registration Authority

Fig 3. Single root with Registration authority

In this design, there is still a single issuing root, however the enrollment requests are handled by an RA (Registration Authority). An RA acts as a proxy between the CA and the endpoint. This allows for the CA to have strict access controls yet still be able to process enrollment requests and CRL downloads.

Offline Root & Issuing Subordinate CAs w/RA

The final variation is a multi-level PKI that uses the RA to process enrollment and CRL downloads. This design provides the best combination of security, scaling, and flexibility, but it is also the most complex.

Bootstrapping remote routers

Consider a situation where you’re turning up a remote router and you need to bring up your transport tunnel to the Datacenter. The Certification authority lives in the Datacenter on a limited access network behind a firewall. In order for the remote router to enroll in-band using SCEP, it would need a VPN connection. But our VPN uses digital certificates.

There are three options for solving this:

Sideband tunnel w/pre-shared key

This method involves setting up a separate VPN tunnel that uses a pre-shared key in order to provide connectivity for enrollment. This could be a temporary tunnel that’s removed when enrollment is complete, or it could be shut down and left in place for use at a later time for other management tasks.

A sideband tunnel for performing management tasks that may bring down the production tunnel(s) is useful, making this a good option. It’s main drawback is the amount of configuration work required on the remote router. It also depends on some expertise on the part of the installer, a shortcoming shared with the manual enrollment method.

Registration Authority

A Registration Authority is a proxy that relays requests between the Certification Authority and the device requesting enrollment. In this method an RA is enabled on the untrusted network for long enough to process the enrollment request. Once the remote router has been enrolled the the transport tunnels will come up and bootstrapping is complete.

Using a proxy allows in-band enrollment with a minimum amount of configuration on the remote router, making it less burdensome for the on-site field technician. The tradeoff is we’re shifting some of that work to the head end. Because the hub site staff is likely to possess more expertise, this is an attractive trade-off.

Manual/Terminal enrollment

In this method, the endpoint produces a Enrollment request on the terminal formatted as base64 PEM (Privacy Encrypted Mail) Blob. The text is copied and pasted into the terminal of the CA. The CA processes the request and outputs the certificate as a PEM file, which is then pasted into the terminal of the Client router.

While this does have the advantage of not requiring network connectivity between the CA and the enrolling router, it does have a couple of drawbacks. Besides being labor intensive and not straightforward for a field technician to work with, endpoints enrolled with the terminal method cannot use the auto-rollover feature, which allows the routers to renew certificates automatically prior to expiration. The author regards this is an option of last resort.

CRL download on spokes problem

The issue here is when the spoke router needs to download the certificate revocation list (CRL) but the CA that has the CRL is reachable only over a VPN tunnel, that cannot come up because the spoke can’t talk to the CA to download a unexpired copy of the CRL in order to validate the certificate of the head end router.

This is actually a pretty easy problem to solve. Disable CRL checking on the spoke routers, but leave it enabled on the hub routers. This way the administrator can revoke a router certificate and that router will not be allowed to join the network because the hub router will see that it’s certificate has been revoked.


Ok, so there are the basics. In the next installment, we’ll step through a minimum working configuration.

Blowing the dust off


Hard to believe it’s been a year since the last update. Time really flies doesn’t it?

I’ve maintained a study schedule, however it’s been oriented around topics that don’t make any sense for a technical blog. That’s going to continue for a time, but I’m also bringing relevant topics back into the mix. This means more regular updates here (yay!).

In addition to straight technical content , I’ll be adding some context and high level overview type things. This is my way of processing and participating in the ongoing community dialog about learning, skill stacks, and general professional development.

There are a lot of exciting things going on and not enough hours in the day. I feel like a kid in a candy store when I get up every morning. Let’s get to stuffing our pockets shall we?

Best regards,


CCIE Security – Why and How

Greetings programs!

I’ve been quite surprised by the response to the news that I passed the lab.  I suspect a lot of it is due to my friend Katherine who has a sizeable following on account of her excellent blog. If you haven’t checked it out, you should!

Anyhow, from all my new friends, I received a lot of questions.  This post is intended to answer them in one place, as well as supply some additional context, and share some of the key resources that I used in my preparation.

The big question, Why?

For large undertaking like this, you need to have a pretty strong reason, or reasons.  It’s a sizeable commitment and when things get really hard,  you’re likely to give up if there’s not a good reason for doing it.

In my case there were several reasons.  Listed in no particular order:

  • Increase my knowledge level. I like to have a deep understanding of what I’m working with.
  • Increase my income earning capability
  • Prove to myself that passing Route/Switch wasn’t a fluke; that I could put my mind to a goal of similar scope and do it again in an area where I have less background.
  • I enjoy the topics.  These toys are a **lot** of fun to play with.

Having seem mine, hopefully it will help you work with defining yours.

The commitment required

Let’s take a realistic assessment of the level of effort required.  Understand this is going to specific to the individual based upon:

  • The existing expertise brought to the starting line
  • How much of the blueprint covers work performed in day job
  • Ability to purchase quality training materials
  • Quality of the structures used to facilitate learning process
  • Consistent application of effort (i.e. not engaging in distractions during study time)
  • Capacity to absorb and process the information

In my case, I was already familiar with much of the VPN technology and I knew my way around ASA firewalls and firepower, all of which I implement and support in my day job. Additionally  I had recently completed CCIE Route/Switch, so I knew how to structure information for later retrieval and how to establish a sustainable study routine. However I was a complete newbie to ISE and Cisco wireless networking.

With all that, it took me approximately 19 months and two attempts, studying an average of 20 hours a week, for a total effort of somewhere around 1600-1700 hours.  I reckon the range for the vast majority of candidates would fall somewhere between 1200 and 2500 hours, presuming they average 20 hours a week of quality study time without significant gaps.

Support and buy in from your family

Consider this:  You’re already employed with full time job  and you have adult obligations to maintain.  When you start studying 20+ hours a week, your wife and your kids (if you have them) are going to feel your absence, and it’s not going to be easy for them.  There needs to be relevant reasons for them as well because they are going to sacrifice alongside you.  It’s a good idea to talk to them and ensure they have a crystal clear understanding of what the next 18 to 24 months going to be like, and get their buy-in.  This is not worth losing your family over!

I strongly recommend taking one day off a week to spend that quality time with your family. That may not be possible in your final push, but that’s just the end phase.  This is a marathon not a sprint.

Establishing milestones

One thing that derails a lot of candidates is lacking a clear path to the end goal.  You need a sense of urgency and momentum. It’s easy to get lost in the technical deep dives (especially with the larger topics like VPN and ISE), but it’s important to keep moving.

It’s ok to make adjustments as you go along, and there will be setbacks.  The idea is to have something to help you stay on track so at some point you’ve got your ass planted in a seat at a lab location and you’re making it happen.

Example high level outline

Months 1-3:  Pass one through the blueprint

  • Focus on basics
  • 70% theory 30% lab

Months 4-7:  Pass two through the blueprint

  • Go deeper into the details and advanced use cases, read rfc and advanced docs.
  • 50% theory 50% lab

Months 8-9:  Prepare for written

  • Heavy on theory and facts
  • Study written only topics
  • 80% theory 20% lab

End of month 9: take written exam

Months 10-12: Pass three through the blueprint

  • Repeat deep dive, with more emphasis on hands on
  • focus mostly on VPN and ISE
  • 30% theory 70% lab

Months 13-14: multi-topic labs

  • No more deep dives
  • Combine multiple technologies in each labbing session
  • get used to shifting gears
  • Wean off cli.  Build all config in notepad
  • 20% theory 80% lab

Schedule lab for 90 days out

Months 15-17: final push for lab

  • Lab every topic on blueprint during the week
  • Lab all topics in one sitting 1x/week
  • all config in notepad
  • speed and accuracy paramount
  • Stop using debugs
  • Time all exercises
    • include verification in time
  • Get friend to break your labs for troubleshooting practice
  • 10% review, 90% lab

End of month 17: First attempt

The battle of memory

Setting up your notebook and files

In the beginning, you’re not going to have much to look at in terms of accumulated knowledge. That will gradually change. It’s a really good idea to set up a system at the beginning; a whole lot easier before you’ve accumulated a bunch of stuff.

My preference these days is OneNote, but other tools will of course work fine.  Heck even an old 3 ring binder can do the job if that’s what you’re into.

Converting the blueprint into something more practical

When your setting up your notebook, the first intuition might be to mirror the blueprint.  i.e. a tab called 1.0 Perimeter Security and Intrusion Protection.   Here’s my suggestion: Organize by product and technology, with additional tabs for unprocessed notes, running and completed tasks, general info, classes taken, etc.  Here’s a screenshot of my tab list for example:


Should you start a blog?

There’s a very effective learning technique called the Feynman Technique. It consists of 5 steps.

  1. Write down the name of a concept
  2. Write out an explanation of the concept as you understand it in plain English.
  3. Review what you’ve written, study the gaps you’ve discovered in your knowledge. Update your explanation
  4. Review your explanation a final time and look for ways to simplify and improve clarity.
  5. Share this explanation with someone.

It really does work.

Blogging is a perfect vehicle for this sort of thing.  But here’s the rub: Blogging is a skill that takes effort to develop.  In my own experience, my early posts took hours to write and I just about gave up on it due to the amount of time that it took.  But as I kept doing it, I got more efficient, and ultimately I found it gave a good return on the time invested.

It’s not a requirement of course.  It’s just a tool, and there lots of other useful tools to choose from.

Written prep and fact memorization Tools

The written covers things you’ll see in the lab, but it also stresses blueprint items that aren’t testable in hands on environments, such as knowledge about theory, standards and operations, and emerging technologies.

You’ll need to have a lot of facts at your recall for the written, so you’re going to have to do a lot of memorization.  That means good old spaced repetition.

The two products that I’ve used to aid in memorization are Anki and Supermemo.  I have a personal preference for supermemo, but it’s a quirky program with a learning curve.  Anki is much easier to pick up and start using.  Either one of these tools are a good aid for the memorization grind that’s part and parcel of passing a CCIE written exam.

Gathering learning resources

This will be an ever expanding list as you get deeper into your studies, but before you start, it makes sense to have some material lined up to take you on that first tour though the blueprint.  As there is not a one stop guide for the security track, this first step can take a little bit of time.  You should have your study material for the first six to 9 months lined up before you dive in order to ensure you make productive use of your time.  The resource list near the end of this post Will hopefully make a good starting point.

Labbing equipment

You will want to invest in some equipment for this track.  If you’re serious about doing this, it’s the best way.  Yes you can do quite a bit on GN3 and EVE-NG, but you’re going to want to have a lot of seat time with the switch and the firewalls, and you’re going to want large topologies that include resource hungry appliances.  A good strategy is to pool your resources with a couple of other people and host your gear in a cheap colo.

This is the lab gear that I recommend:

  • Server: EXSi host with 128gb ram, 12 cores, 1tb disk
  • Switch: Catalyst 3650 or 3850 is best, a 3750x is a good budget option
  • AP:  3602i or similar
  • IP Phone: Cisco 7965G
  • Firewall: 2xASA 5512-X with clustering enabled.
  • Optional gear for COLO:
    • Terminal Server (I Prefer OpenGear IM series: not cheapest but quite secure)
    • Edge firewall:  ASA 5506x

Equipment thoughts and notes

There’s lots of good deals on older 1u rack mount servers on ebay, that’s easy information to research.  If you can afford more than 128gb of ram, go for it.  You’ll use all of it once you start labbing larger toplogies.

When shopping for switches on ebay avoid switches with the LAN Base image, as IP Base is the minimum version required to use most features on the blueprint.  Also, it never hurts to check the Trustsec Compatibility Matrix for the specific part you’re looking at buying.

Yes, you want to drop the coin on a pair of ASAs.  Just do it. You don’t have to spring for the x series, but you do need multicontext and clustering, and you need to be able to run version 9.2 of the ASA software, those are your minimums.

My labbing partner and I started out with a cheap terminal server and the darn thing kept getting DOSed, so we dropped 400 bucks on an older Opengear terminal server, and that thing is bulletproof.  Worth the extra money for out of band access to your gear.

Most COLOs, even budget ones, have remote hands to power cycle things for you if they get locked up.  But if you’re not comfortable with that, hooking up a PDU like an old APC AP7931 to your terminal controller will work.  I use one of these for my home rack and they’re great.

About software and platform versions

You want to match the versions on the blueprint as much as you can. Where it’s not realistic or a hassle to do so; I strongly urge you to check your syntax and configs against the lab version periodically if possible.

Personal experience: I my environment I labbed on the newer version of the CSR1kv image that has the 1mb/sec throughput limitation in demo mode.  That’s great, but some of my configuration speed optimizations blew up on lab day.  That repeated in several areas, including differences between the 3750x in my lab and the 3850 in the actual lab.

Bottom line:  It’s a lot of money and stress to go sit the lab, and when the clock is ticking and you’re running into unexpected platform and software issues, the little bit of money you saved on that screaming ebay deal for an older part will be long forgotten.

About licensing the appliances in your lab.

Most of stuff is not a problem, you can run them on demo licenses. The exceptions are NGIPS WSA and ESA which are a pain.  If you don’t work for a partner or Cisco, your local Cisco SE should be able to get you 3 month licenses for those appliances to assist with lab prep.

Establishing the daily routine

This is a critical item. It’s just like a fitness routine.  Consistency is everything.  You’re not going to be able skip studying during the week and binge on weekends expecting quality results.  It just doesn’t work that way.

In my view, the bare minimum do to this in 18 months is 20 hours a week.  That’s 20 focused hours without distractions.

There are going to be days when you *really* don’t want to study. Embrace the suck my friend, and just do it anyway.

Suggested Schedule

  • 3-4 hours a day Monday through Friday
  • 8 hours Saturday
  • Sunday off

If you stick to that week in and week out, you will make good steady progress.

One thing that really helped me is a tool called tomato timer.  The idea is for 25 minutes you’re hyper focused.  Then you take 5 minutes off. Then 25 minutes on again.  It really helps with procrastination and the urge to check your twitter feed and all that.

Responding when life gets in the way

FACT:  Things are going to happen that are out of your control.  You’re probably going to get pulled away from your studies for a week or two here and there.

Try to at least read a little bit and stay mentally engaged on some level every day until the clouds part and you’re able to get back on the grind again.  Main thing is don’t let a short break turn into a long one; it happens very easily and you’ll be upset with yourself as you have to retrace your steps to get back to where your were in your progress.

Resource List:

Here’s a list of learning resources that I found to be particularly helpful.  (Not including obvious things like product documentation).



Instructor led Training


DCloud Labs

  • Security Everywhere
  • IPSEC VPN troubleshooting
  • Web Security Appliance Lab v1

Cisco Live Sessions

Cisco ISE community

Closing thoughts

I hope this answered some of your questions and gave you a sense of a path to get to the CCIE Security lab.  I really enjoyed reliving all the memories creating this post triggered.  It’s been quite an adventure. 🙂

Best regards, and may you care for yourself with ease,