Skip to content

Home

Jump to bottom
Claudia de Luna edited this page May 2, 2020 · 19 revisions

Welcome to the pandas_neteng wiki!

For the full write up on this repository please see "Pandas for Network Engineers (Who doesn’t love Pandas?)"

The write up below provides an overview of one of the scripts in the repository (simple_mac_compare.py) which uses Pandas to compare MAC Address Tables. The specific scenario illustrated below is a Data Center migration from a traditional NX-OS environment into Cisco ACI.

The script will:

  • take show command output from a Cisco NX-OS switch and
  • take the results of a REST call to the APIC,
  • parse and normalize that data, and
  • use Pandas to quickly compare so we can easily determine if all of our Macs (servers) from the NX-OS environment are now on the ACI fabric.

¡Aviso! Stay tuned for an enhanced and simplified version of this using pyATS Genie!

Overview of simple_mac_compare.py script

For this script you only need the pandas module installed. I've taken some shortcuts to illustrate the comparison aspect of this. In production scripts like this one, you will use various other Python modules (TextFSM, requests, nornir, netmiko, and others) to get the data to compare. Here I wanted to focus on the comparison aspects with a minimal virtual environment (Python and Pandas).

pip install pandas

If you have cloned the full repository and set up the required virtual environment you should be good to go.

The current script works on static data but its set up to easily replace the static data functions with real time data gathering.

  • From the legacy NX-OS environment the output of the show mac address-table command form an NX-OS device
  • From ACI the REST JSON response to GET https://{{URL}}/api/node/class/fvCEp.json

The "load" functions in the script which load the static data can be converted to functions which load data from a text file (the file you created and saved when you ran the show mac address-table command on your NX-OS devices) and perform a real time query on the APCI via REST.

However you get it, the NX-OS show command output is loaded and then manipulated (parsed) to extract vlan and MAC data (the NX-OS parsing grabs a few more values). The data is manipulated to form a list of lists for each entry in the mac table. In this data structure it is easy to iterate through each item representing a MAC address and its characteristics and apply logic.

For this exercise, we are working to answer the question: "Is this NX-OS mac in the ACI mac list?"

With Python it is a simple matter to write a bit of logic to check each NX-OS MAC is in the ACI list.

However, it is also very easy to make a Pandas data frames from each list.

Once we have data frames from each set of data, we add to each data frame a new column of normalized MACs so we can do comparisons. The normalize_mac function takes in a MAC address in any format and strips out any punctuation, returning just the numbers and letters in lower case. Tip: In a production environment I would use the netaddr module for this task but for this exercise we are working with a minimalist environment (Python and Pandas) hence this "string substitution" approach.

def normalize_mac(mac):
    """
    Given a Mac, strip periods, colons, or dashes from the string and return result in lower case
    :param mac:
    :return:
    """
    return re.sub(r'(\.|:|\-)', '', mac).lower()

Now each data frame has a column titled 'normalized_mac' with MACs in the same format which can be used for comparison.

NXOS Data Frame: 
    0    1               2        3 normalized_mac
0   *   98  0008.e3ff.fd8c  dynamic   0008e3fffd8c
1   *   98  0050.5684.5b01  dynamic   005056845b01
2   *   99  0000.0000.0100  dynamic   000000000100
3   *  750  a025.b5f2.5000  dynamic   a025b5f25000
4   *  750  a025.b5f2.5004  dynamic   a025b5f25004
5   *  704  0008.e3ff.fd8c  dynamic   0008e3fffd8c
6   *  700  0050.5680.5f61  dynamic   005056805f61
7   *  700  0050.5680.6564  dynamic   005056806564
8   *  700  0050.5684.5f2f  dynamic   005056845f2f
9   *  700  0050.5684.cca7  dynamic   00505684cca7
10  *  700  0050.5684.d66e  dynamic   00505684d66e
11  *  700  80e0.1d37.1e18  dynamic   80e01d371e18
12  *  700  80e0.1d37.1e1e  dynamic   80e01d371e1e
13  +  700  80e0.1d37.2b7c  dynamic   80e01d372b7c
14  *  700  80e0.1d37.2b82  dynamic   80e01d372b82
15  +  700  e4aa.5dac.81f6  dynamic   e4aa5dac81f6
16  *  700  e4aa.5dac.81f7  dynamic   e4aa5dac81f7
17  *  699  0008.e3ff.fd8c  dynamic   0008e3fffd8c
18  *  699  02a0.98d3.71f5  dynamic   02a098d371f5

ACI Data Frame: 
                   0         1 normalized_mac
0  42:5D:BC:C4:00:00  vlan-123   425dbcc40000
1  42:5D:BC:C4:00:00  vlan-121   425dbcc40000
2  42:5D:BC:C4:00:00  vlan-125   425dbcc40000
3  42:5D:BC:C4:00:00  vlan-127   425dbcc40000
4  00:50:56:80:65:64  vlan-700   005056806564

This approach lets us leverage the Pandas merge method so that we can easily merge the two data frames and determine what the difference are and where they came from.

The merged data frame below shows the combined data frame along with where the data came from in the "Exist" column. Now it is very easy to tell which Macs originated in the NXOS data and which are in ACI and which are in both! This can be done in one command!

df_merged = pd.merge(df_nxos, df_aci, on='normalized_mac', how='outer', indicator='Exist')

We build a new data frame called "df_merged". The first argument is the "left" data frame (df_nxos) with the NX-OS data and the second argument (df_aci) is the "right" data frame with the ACI data. We are merging on the 'normalized_mac' column and essentially doing an outer join for those familiar with databases using the 'normalized_mac' column as keys. As if this was not good enough, the "indicator='Exists'" parameter will add a column (Exists) to the new resulting data frame telling us where the data came from (and if data was found in both data sets). In this small data frame its very easy to see that only one NX-OS MAC is on ACI.

NX-OS and ACI MERGED Data Frame: 
    0_x  1_x               2        3 normalized_mac                0_y       1_y       Exist
0     *   98  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
1     *  704  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
2     *  699  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
3     *   98  0050.5684.5b01  dynamic   005056845b01                NaN       NaN   left_only
4     *   99  0000.0000.0100  dynamic   000000000100                NaN       NaN   left_only
5     *  750  a025.b5f2.5000  dynamic   a025b5f25000                NaN       NaN   left_only
6     *  750  a025.b5f2.5004  dynamic   a025b5f25004                NaN       NaN   left_only
7     *  700  0050.5680.5f61  dynamic   005056805f61                NaN       NaN   left_only
8     *  700  0050.5680.6564  dynamic   005056806564  00:50:56:80:65:64  vlan-700        both
9     *  700  0050.5684.5f2f  dynamic   005056845f2f                NaN       NaN   left_only
10    *  700  0050.5684.cca7  dynamic   00505684cca7                NaN       NaN   left_only
11    *  700  0050.5684.d66e  dynamic   00505684d66e                NaN       NaN   left_only
12    *  700  80e0.1d37.1e18  dynamic   80e01d371e18                NaN       NaN   left_only
13    *  700  80e0.1d37.1e1e  dynamic   80e01d371e1e                NaN       NaN   left_only
14    +  700  80e0.1d37.2b7c  dynamic   80e01d372b7c                NaN       NaN   left_only
15    *  700  80e0.1d37.2b82  dynamic   80e01d372b82                NaN       NaN   left_only
16    +  700  e4aa.5dac.81f6  dynamic   e4aa5dac81f6                NaN       NaN   left_only
17    *  700  e4aa.5dac.81f7  dynamic   e4aa5dac81f7                NaN       NaN   left_only
18    *  699  02a0.98d3.71f5  dynamic   02a098d371f5                NaN       NaN   left_only
19  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-123  right_only
20  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-121  right_only
21  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-125  right_only
22  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-127  right_only

With more data it will be easier to query the data frame for just the information we need.
How many of my legacy MACs from my NX-OS switch are now on ACI?

found_df = df_merged.loc[df_merged['Exist'] == 'both']

Found 1 MAC(s): 
  0_x  1_x               2        3 normalized_mac                0_y       1_y Exist
8   *  700  0050.5680.6564  dynamic   005056806564  00:50:56:80:65:64  vlan-700  both

Which MACs are missing?

notfound_df = df_merged.loc[df_merged['Exist'] != 'both']

22 ALL MAC(s) MISSING: 
    0_x  1_x               2        3 normalized_mac                0_y       1_y       Exist
0     *   98  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
1     *  704  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
2     *  699  0008.e3ff.fd8c  dynamic   0008e3fffd8c                NaN       NaN   left_only
3     *   98  0050.5684.5b01  dynamic   005056845b01                NaN       NaN   left_only
4     *   99  0000.0000.0100  dynamic   000000000100                NaN       NaN   left_only
5     *  750  a025.b5f2.5000  dynamic   a025b5f25000                NaN       NaN   left_only
6     *  750  a025.b5f2.5004  dynamic   a025b5f25004                NaN       NaN   left_only
7     *  700  0050.5680.5f61  dynamic   005056805f61                NaN       NaN   left_only
9     *  700  0050.5684.5f2f  dynamic   005056845f2f                NaN       NaN   left_only
10    *  700  0050.5684.cca7  dynamic   00505684cca7                NaN       NaN   left_only
11    *  700  0050.5684.d66e  dynamic   00505684d66e                NaN       NaN   left_only
12    *  700  80e0.1d37.1e18  dynamic   80e01d371e18                NaN       NaN   left_only
13    *  700  80e0.1d37.1e1e  dynamic   80e01d371e1e                NaN       NaN   left_only
14    +  700  80e0.1d37.2b7c  dynamic   80e01d372b7c                NaN       NaN   left_only
15    *  700  80e0.1d37.2b82  dynamic   80e01d372b82                NaN       NaN   left_only
16    +  700  e4aa.5dac.81f6  dynamic   e4aa5dac81f6                NaN       NaN   left_only
17    *  700  e4aa.5dac.81f7  dynamic   e4aa5dac81f7                NaN       NaN   left_only
18    *  699  02a0.98d3.71f5  dynamic   02a098d371f5                NaN       NaN   left_only
19  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-123  right_only
20  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-121  right_only
21  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-125  right_only
22  NaN  NaN             NaN      NaN   425dbcc40000  42:5D:BC:C4:00:00  vlan-127  right_only

Really we just care about the NX-OS MACs that are not in ACI.

nxosnotfound_df = df_merged.loc[df_merged['Exist'] == 'left_only']

18 NXOS MAC(s) MISSING from ACI: 
   0_x  1_x               2        3 normalized_mac  0_y  1_y      Exist
0    *   98  0008.e3ff.fd8c  dynamic   0008e3fffd8c  NaN  NaN  left_only
1    *  704  0008.e3ff.fd8c  dynamic   0008e3fffd8c  NaN  NaN  left_only
2    *  699  0008.e3ff.fd8c  dynamic   0008e3fffd8c  NaN  NaN  left_only
3    *   98  0050.5684.5b01  dynamic   005056845b01  NaN  NaN  left_only
4    *   99  0000.0000.0100  dynamic   000000000100  NaN  NaN  left_only
5    *  750  a025.b5f2.5000  dynamic   a025b5f25000  NaN  NaN  left_only
6    *  750  a025.b5f2.5004  dynamic   a025b5f25004  NaN  NaN  left_only
7    *  700  0050.5680.5f61  dynamic   005056805f61  NaN  NaN  left_only
9    *  700  0050.5684.5f2f  dynamic   005056845f2f  NaN  NaN  left_only
10   *  700  0050.5684.cca7  dynamic   00505684cca7  NaN  NaN  left_only
11   *  700  0050.5684.d66e  dynamic   00505684d66e  NaN  NaN  left_only
12   *  700  80e0.1d37.1e18  dynamic   80e01d371e18  NaN  NaN  left_only
13   *  700  80e0.1d37.1e1e  dynamic   80e01d371e1e  NaN  NaN  left_only
14   +  700  80e0.1d37.2b7c  dynamic   80e01d372b7c  NaN  NaN  left_only
15   *  700  80e0.1d37.2b82  dynamic   80e01d372b82  NaN  NaN  left_only
16   +  700  e4aa.5dac.81f6  dynamic   e4aa5dac81f6  NaN  NaN  left_only
17   *  700  e4aa.5dac.81f7  dynamic   e4aa5dac81f7  NaN  NaN  left_only
18   *  699  02a0.98d3.71f5  dynamic   02a098d371f5  NaN  NaN  left_only


============== SUMMARY ============== 
Of 23 Total MACs both legacy and ACI: 
        - NX-OS FOUND IN ACI            1 
        - TOTAL MACs MISSING            22
        - NX-OS MISSING FROM ACI        18
Clone this wiki locally