netcrawl package


netcrawl.cli module

Created on Feb 28, 2017

@author: Wyko

netcrawl.cli.connect(handler=None, netmiko_platform=None, ip=None, cred=None, port=None)[source]

Starts a CLI session with a remote device.

Uses Netmiko to start a SSH or Telnet session with a target device. It will attempt to use SSH first, and if it fails it will try Telnet. For each connection method, it will attempt each credential specified in the cred argument (if specified) or the list otherwise.

Keyword Arguments:
  • cred (dict) – If supplied, this method will only use the specified credential. Uses the list otherwise.
  • port (int) – If supplied, this method will connect only on this port
  • ip (str) – The IP address to connect to
  • netmiko_platform (str) – The platform of the device, in the Netmiko format
  • handler (ConnectHandler) – A Netmiko-type handler to use. Currently using one of Netmiko.ConnectHandler, Netmiko.ssh_autodetect.SSHDetect. Uses Netmiko.ConnectHandler by default.

A dict containing:

  • connection (ConnectHandler): A Netmiko ConnectHandler object with a successfully opened connection
  • tcp_22 (bool): True if port 22 is open
  • tcp_23 (bool): True if port 23 is open
  • username (str): The first successful credential’s username
  • password (str): The first successful credential’s password
  • cred_type (str): The first successful credential’s type

Return type:


  • IOError – If a connection could not be established
  • AssertionError – If error checking failed

netcrawl.config module

class netcrawl.config.Config[source]

Bases: object

set_all_database_creds(username, password)[source]
class netcrawl.config.Database(dbname)[source]

Bases: object


Returns a dict used to populate a psycopg2 connection


netcrawl.core module

netcrawl.core.nmap_scan(target, **kwargs)[source]

Ping each host in a given range one at a time. When a live host is found, add it to the pending hosts database.

Parameters:target (str) – An Nmap compatible target specifier as outlined in the Nmap documentation
Keyword Arguments:
 **kwargs – Arguments to pass to netcrawl.io_sql.main_db

Prints a brief report of the state of the databases to the console


Starts a Recursive Scan (-sR) run. This is the main scanning method for netcrawl.

1. If a target kwarg is given, add that seed device to the list of pending deivces, even if it was already visited.

2. Create workers (subprocesses) to perform the scanning work, up to 16 per CPU core, or up to the processes kwarg per core if that kwarg was given.

3. Query the Pending table in the Main database for pending devices.

4. Autodetect the Netmiko platform for each device if needed.

5. Inventory the device using netcrawl.devices.base.NetworkDevice.process_device()

  1. Add each discovered device to the Inventory database
Keyword Arguments:
  • skip_named_duplicates (bool) –

    If True, this will cause netcrawl to skip neighbors which have the same hostname as a device that was previously visited.


    While this can potentially save a lot of time when scanning devices, if multiple different devices share the same hostname, they will not be scanned!

  • target (str) – The IP address of a seed device to add to the pending devices database
  • netmiko_platform (str) – The Netmiko platform of the target device, if one was given.
  • processes (int) – The number of worker processes to create, multiplied by the CPU count


If there are any remaining keyword arguments in **kwargs, they will be passed to netcrawl.io_sql.main_db and netcrawl.io_sql.device_db

netcrawl.core.single_scan(target, netmiko_platform='unknown')[source]

Starts a Single Scan (-sS) run. This scan polls a single device and presents information about the device to the console. Useful for testing a connection, as well as getting a quick overview of the target.

Keyword Arguments:
  • target (str) – The network address of the device to scan
  • netmiko_platform (str) – The Netmiko platform of the target device. If one is not given, it will attempt to autodetect the device type.

netcrawl.device_dispatcher module

Controls selection of proper class based on the device type.

Credit: Kirk Byers


This method invokes Netmiko’s autodetect functionality to determine the correct device class, then returns that class as a netmiko_platform.


target (String) – The hostname or IP address to connect to

  • TypeError – Could not find an appropriate class to inherit
  • IOError – Could not connect to the device

The netmiko_platform representation of the proper

device class.

Return type:


netcrawl.device_dispatcher.create_instantiated_device(*args, **kwargs)[source]

Factory function selects the proper network device class and creates the object based on netmiko_platform.

netcrawl.io_sql module

class netcrawl.io_sql.device_db(**kwargs)[source]

Bases: netcrawl.io_sql.sql_database


Appends a device to the database

Parameters:_device (network_device) – A single network_device
Returns:False if write was unsuccessful Int: Index of the device that was added, if successful
Return type:Boolean
delete_device_record(id, cur=None)[source]

Removes a record from the devices table

exists(device_id=None, unique_name=None, device_name=None)[source]

Checks whether a device record is present in the devices table. Tries each supplied identifier in order until a match is found, then returns the device_id of the found record.

Keyword Arguments:
  • device_id (int) – If not None, check the device_id column for a match
  • unique_name (str) – If not None, check the unique_name column for a match
  • device_name (str) – If not None, check the device_name column for a match
int: The device_id of the first match found bool: False if not found
get_device_record(column, value)[source]

Get a device record based on a lookup column. ‘WHERE column = value’

Returns:psycopg2 dict object
insert_device_entry(device, cur)[source]
insert_interface_entry(device_id, interf, cur)[source]
insert_mac_entry(device_id, interface_id, mac_address, cur)[source]
insert_neighbor_entry(device_id, interface_id, neighbor, cur)[source]
insert_neighbor_ip_entry(neighbor_id, ip, cur)[source]
insert_serial_entry(device_id, serial, cur)[source]
locate_mac(mac, cur=None)[source]

Parent method for handling an existing device which needs to be updated.

  1. Determine if the device exists and, if so, get the device_id
  2. Overwrite all entries in the device with the new device
  3. Set a new updated time for all dependent tables
  4. Delete any interfaces and serials which no longer exist
  5. Add any new interfaces and serials
  6. Add any new MAC addresses
  7. Update any newly non-existent MAC addresses
Parameters:device (NetworkDevice) – A network device object to check against for duplicates
Returns:True if a duplicate was found and updated
Return type:bool
set_dependents_as_updated(device_id, cur=None)[source]

Sets the last touched time on all dependents of the given device_id to now

update_device_entry(device, cur=None, device_id=None, unique_name=None)[source]

Overwrites all entries in the Devices table with a matching device_id or unique_name with the information in device.


device (NetworkDevice) – The device to source updates from

Keyword Arguments:
  • cur (psycopg2.cursor) – Cursor object used to update the database
  • device_id (int) – If not None, overwrites the row at this index with device
  • unique_name (str) – If not None, overwrites any row with a matching unique_name field with device.


If both device_id and unique_name are given, the method will update all entries that match either key.

Raises:ValueError – No unique_name or device_id passed to the method
class netcrawl.io_sql.main_db(**kwargs)[source]

Bases: netcrawl.io_sql.sql_database

add_device_pending_neighbors(_device=None, _list=None)[source]

Appends a device or a list of devices to the database

Optional Args:
_device (network_device): A single device _list (List): List of devices
Returns:True if write was successful, False otherwise.
Return type:Boolean
add_pending_device_d(device_d=None, cur=None, **kwargs)[source]
add_visited_device_d(device_d=None, cur=None, **kwargs)[source]
add_visited_device_nd(_device=None, _list=None, cur=None)[source]

Appends a device or a list of devices to the database

Optional Args:
_device (network_device): A single network_device _list (List): List of network_device objects
Returns:True if write was successful, False otherwise.
Return type:Boolean

Counts the number of rows in the table


Counts the number of unique devices in the database


Gets the next pending device.

The next pending device as a dictionary object
with the names of the rows as keys.
Return type:Dict

Removes a record from the pending table


Removes a record from the pending table

class netcrawl.io_sql.sql_database(**kwargs)[source]

Bases: object

A base class to facilitate SQL database operations.

Keyword Arguments:
 clean (bool) –

If True, this causes all database tables to be dropped in order to start with a clean database.


Obviously, this is really dangerous.


Closes the connection to the database if it is open

count(table, column='*', value=None, partial_value=None, distinct=False, cur=None)[source]

Counts the occurrences of the specified column in a given table.


table (str) – The table to search in

Keyword Arguments:
  • column (str) – The column to count
  • distinct (bool) – If True, count only unique matches
  • value (str) –

    If not None, adds a where clause to the count in the format:

    WHERE column = 'value'
  • partial_value (str) –

    If not None, adds a where clause which will match a partial string in the format:

    WHERE column like '%partial_value1%'

The number of matches

Return type:



Creates a new database

Parameters:new_db (str) – Database name to create

Returns true is the specified database exists

Parameters:db (str) – A database name
Returns:True if the database exists
Return type:bool

Deletes a database


True if the database was created

Return type:


  • FileExistsError – If the database to be deleted does not exist.
  • IOError – If the database could not be deleted and still exists after execution
execute_sql(*args, proc=None, fetch=True)[source]

Executes a SQL snippet and optionally gets the results


*args – The arguments to pass along to pyscopg2.cursor.execute(). Usually a string containing the SQL statement, and potentially a tuple of parameters.

Keyword Arguments:
  • proc (str) – The name of the parent process, for logging purposes
  • fetch (bool) – If True, fetches all results from the query

The results of pyscopg2.cursor.fetchall()

Return type:


execute_sql_gen(*args, proc=None)[source]

Executes a SQL snippet and gets the results in a generator

Parameters:*args – The arguments to pass along to pyscopg2.cursor.execute(). Usually a string containing the SQL statement, and potentially a tuple of parameters.
Keyword Arguments:
 proc (str) – The name of the parent process, for logging purposes
Returns:The results of pyscopg2.cursor.fetchall()
Return type:generator
ip_exists(ip, table)[source]

Check if a given IP exists in the database

  • ip (str) – The IP address to check for
  • table (str) – The table to check

ValueError – If an argument is an improper type

ip_name_exists(ip, name, table, cur=None)[source]

Check if a given IP OR Name exists in the database

class netcrawl.io_sql.sql_logger(proc, ignore_duplicates=True)[source]

Bases: object

Utility class to enable logging and timing SQL execute statements, as well as handling specific errors


Decorator that creates a cursor object to pass to the wrapped method in case one wasn’t passed originally. The wrapped function should accept cur as an argument.

netcrawl.util module

class netcrawl.util.benchmark(name)[source]

Bases: object

Context manager which times the surrounded code and prints the results to the console


Changes CIDR notation to subnet masks. I honestly have no idea how this works. I just added some error checking.

class netcrawl.util.cleanExit[source]

Bases: object

Context manager who’s only purpose is to cleanly exit when the code execution is interrupted by the user


Removes all non-digit or period characters from the source string


Simple boolean operator to determine if a string contains a mac anywhere within it.


Get stored credentials using a the credentials module. Requests credentials via prompt otherwise.

Returns:[{username, password, cred_type}, ]

If the username and password had to be requested, the list will only have one entry.

Return type:List of Dicts

Returns true if the given string is an IPv4 address


Translates a netmask to a CIDR format

Parameters:netmask (str) – A netmask in four octet ip address format
Returns:The CIDR representation of the netmask
Return type:int
netcrawl.util.network_ip(ip, subnet)[source]

Returns the network IP address calculated from the given ip and subnet.


Returns a list of strings containing each IP address matched in the input string.

netcrawl.util.port_is_open(port, address, timeout=5)[source]

Checks a socket to see if the specified port is open.

  • port (int) – The numbered TCP port to check
  • address (str) – The address of the host to check
Keyword Arguments:

timeout (int) – The number of seconds to wait before timing out. Defaults to 5 seconds. Zero seconds disables timeout


True if the port is open

Return type:



Decorator method which times the wrapped method and prints the result to the console


Returns the input string stripped of everything but letters, numbers, and underscores.

Module contents