Argos: API Documentation

Constants defined for Argos tools

class argos.constants.Change(frame, end, change, orig, new, idx)

A change action by the user when revieweing tracks. It is defined by the following attributes:

frame

frame number on which the change was applied.

end

frame number till (inclusive) which the change should be applied. -1 for till end of the video.

change

change code, defined in ChangeCode enum

orig

original trackid

new

new trackid

idx

index of change within same frame idx allows maintaining sequence of changes defined within same frame.

property change

Alias for field number 2

property end

Alias for field number 1

property frame

Alias for field number 0

property idx

Alias for field number 5

property new

Alias for field number 4

property orig

Alias for field number 3

class argos.constants.ChangeCode(value)

Code for user defined track changes. These are:

In this and all future frames:

op_swap

Swap IDs (new becomes orig and orig becomes new).

op_assign

Assign new ID to orig ID.

op_delete

Delete orig (new not required)

op_merge

Merge new into orig, this is kept for possible future extension.

Same as above, but apply only in the current frame:

op_swap_cur

op_assign_cur

op_delete_cur

class argos.constants.ColorMode(value)

Coloring scheme to be used when drawing object boundaries and IDs.

single

A single color for all objects. Selected objects in a different color.

cmap

Pick a set of colors from a colormap and rotate.

auto

Pick a random color for each object.

class argos.constants.DistanceMetric(value)

Distance metrics.

iou

Intersection over Union, this is a very common metric where the area of intersection of two patches is divided by their union, producing 0 for no-overlap and 1 for complete overlap.

euclidean

Euclidean distance between objects (usually their centres).

ios

Intersection over smaller. A slight variation of IoU to account for the fact that a small object may be completely overlapping a large object, and yet the IoU will be very small. In our segmentation, we want to merge such objects, and a large IoS suggests merge.

class argos.constants.DrawingGeom(value)

The kind of geometry we are drawing.

A rectangle is defined by (x, y, w, h) but a polygon is a list of vertices (a rectangle could also be defined with this, but the drawing algorithm would be different).

Arena is a special case, when we want to change the visible/processed area of an image, not just draw a polygon on it.

rectangle

Rectangle (use (x, y, w, h) format)

polygon

Polygon defined by sequence of vertices like ((x0, y0), (x1, y1), (x2, y2), ...)

arena

A special case of polygon for defining the area of interest in the image.

argos.constants.EXTRACT_FRAMES = 200

number of frames to extract by default

class argos.constants.OutlineStyle(value)

Outline styles for objects.

bbox

Axis-aligned bounding box.

minrect

Minimum area bounding rectangle, this is a tighter rectangle than bbox, and can be at an angle.

contour

Contour of the object.

fill

Color-filled contour of the object

argos.constants.STYLE_SHEETS = ['default', 'light', 'dark']

styles

class argos.constants.SegStep(value)

Intermediate steps when doing image segmentation using classical image processing methods.

blur

Blurring.

threshold

Thresholding of blurred image.

segmented

Segmentation of the blurred and thresholded image.

filtered

After filtering the segmented objects based on area, width, height, etc.

final

Final segmentation. This is a placeholder to avoid showing any separate window for the results of the intermediate steps.

class argos.constants.SegmentationMethod(value)

Segmentation methods.

threshold

Use thresholding and then bounding boxes of the blobs.

contour

Use thresholding and then filled-contours of the blobs.

dbscan

Use thresholding and then spatially cluster the non-zero pixels with DBSCAN algorithm.

watershed

Use watershed algorithm for segmentation.

class argos.constants.TrackState(value)

Possible tracking states of an object.

argos.constants.change_name = {<ChangeCode.op_assign: 3>: 'assign', <ChangeCode.op_merge: 7>: 'merge', <ChangeCode.op_swap: 1>: 'swap', <ChangeCode.op_delete: 5>: 'delete', <ChangeCode.op_assign_cur: 4>: 'assign at current', <ChangeCode.op_swap_cur: 2>: 'swap at current', <ChangeCode.op_delete_cur: 6>: 'delete at current'}

Dict mapping the change codes to their human readable names.

General utility functions

class argos.utility.WaitingSpinner(centerOnParent=True, disableParentWhenSpinning=True, *args, **kwargs)

paintEvent(self, QPaintEvent)

argos.utility.cond_bbox_overlap(ra, rb, min_iou)

Check if IoU of axis-aligned bounding boxes overlap.

Parameters

• ra (array like) – Rectangles specified as (x, y, w, h)

• rb (array like) – Rectangles specified as (x, y, w, h)

• min_iou (flat) – Minimum value of IoU to consider overlap.

Returns

True if ra and rb have IoU >= min_iou. False otherwise.

Return type

bool

argos.utility.cond_minrect_overlap(ra, rb, min_iou)

Check if IoU of minimum area (rotated) bounding rectangles is at least min_iou.

Parameters

• ra (array like) – First rectangle defined by the coordinates of four corners.

• rb (array like) – Second rectangle defined by the coordinates of four corners.

• min_iou (float) – Minimum overlap defined by intersection over union of bounding boxes.

Returns

True if area of overlap is greater or equal to min_iou.

Return type

bool

argos.utility.cond_proximity(points_a, points_b, min_dist)

Check if the proximity of two arrays of points is more than min_dist.

To take the shape of the object into account, I use the following measure of distance: scale the distance between centres of mass by the geometric mean of the square roots of the second moments.

(x1 - x2) / sqrt(sigma_1_x * sigma_2_x) (y1 - y2) / sqrt(sigma_1_y * sigma_2_y)

Parameters

• points_a (array like) – Sequence of points

• points_b (array like) – Sequence of points

• min_dist (float) – Minimum distance.

Returns

True if the centres of mass (mean position) of points_a and points_b are closer than min_dist, False otherwise.

Return type

bool

argos.utility.cv2qimage(frame: numpy.ndarray, copy: bool = False) → PyQt5.QtGui.QImage

Convert BGR/gray/bw frame from array into QImage”.

OpenCV reads images into 2D or 3D matrix. This function converts it into Qt QImage.

Parameters

• frame (numpy.ndarray) – Input image data as a 2D (black and white, gray() or 3D (color, OpenCV reads images in BGR instead of RGB format) array.

• copy (bool, default False) – If True Make a copy of the image data.

Returns

Converted image.

Return type

qg.QImage

argos.utility.extract_frames(vidfile, nframes, scale=1.0, outdir='.', random=False)

Extract nframes frames from vidfile into outdir

argos.utility.get_cmap_color(num, maxnum, cmap)

Get rgb based on specified colormap cmap for index num where the total range of values is (0, maxnum].

Parameters

• num (real number) – Position into colormap.

• maxnum (real number) – Normalize num by this value.

• cmap (str) – Name of colormap

Returns

tuple – The red, green and blue value for the color at position num/maxnum in the (0, 1) range of the colormap.

Return type

(r, g, b)

argos.utility.init()

Initialize logging and Qt settings.

argos.utility.make_color(num: int) → Tuple[int]

Create a random color based on number.

The provided number is passed through the murmur hash function in order to generate bytes which are somewhat apart from each other. The three least significant byte values are taken as r, g, and b.

Parameters

num (int) – number to use as hash key

Returns

(r, g, b) values

Return type

bytes[3]

argos.utility.match_bboxes(id_bboxes: dict, new_bboxes: numpy.ndarray, boxtype: argos.constants.OutlineStyle, metric: argos.constants.DistanceMetric = DistanceMetric.euclidean, max_dist: float = 10000) → Tuple[Dict[int, int], Set[int], Set[int]]

Match the rectangular bounding boxes in new_bboxes to the closest object in the id_bboxes dictionary.

Parameters

• id_bboxes (dict[int, np.ndarray]) – Mapping ids to bounding boxes

• new_bboxes (np.ndarray) – Array of new bounding boxes to be matched to those in id_bboxes.

• boxtype ({OutlineStyle.bbox, OutlineStyle.minrect}) – Type of bounding box to match.

• max_dist (int, default 10000) – Anything that is more than this distance from all of the bboxes in id_bboxes are put in the unmatched list

• metric ({DistanceMetric.euclidean, DistanceMetric.iou}) – DistanceMetric.euclidean for Euclidean distance between centers of the boxes. DistanceMetric.iou for area of inetersection over union of the boxes,

Returns

• matched (dict[int, int]) – Mapping keys in id_bboxes to bbox indices in new_bboxes that are closest.

• new_unmatched (set[int]) – Set of indices into bboxes that did not match anything in id_bboxes

• old_unmatched (set[int]) – Set of keys in id_bboxes whose corresponding bbox values did not match anything in bboxes.

argos.utility.pairwise_distance(new_bboxes: numpy.ndarray, bboxes: numpy.ndarray, boxtype: argos.constants.OutlineStyle, metric: argos.constants.DistanceMetric) → numpy.ndarray

Computes the distance between all pairs of rectangles.

Parameters

• new_bboxes (np.ndarray) – Array of bounding boxes, each row as (x, y, w, h)

• bboxes (np.ndarray) – Array of bounding boxes, each row as (x, y, w, h)

• boxtype ({OutlineStyle.bbox, OulineStyle.minrect}) – OutlineStyle.bbox for axis aligned rectangle bounding box or OulineStyle.minrect for minimum area rotated rectangle

• metric ({DistanceMetric.euclidean, DistanceMetric.iou}) – When DistanceMetric.euclidean, the squared Euclidean distance is used (calculating square root is expensive and unnecessary. If DistanceMetric.iou, use the area of intersection divided by the area of union.

Returns

Row ii, column jj contains the computed distance between new_bboxes[ii] and bboxes[jj].

Return type

np.ndarray

argos.utility.points2rect(p0: numpy.ndarray, p1: numpy.ndarray) → numpy.ndarray

Convert diagonally opposite vertices into (x, y, w, h) format rectangle.

Returns

Rectangle with diagonal corners p0 and p1 after scaling by scale. This will work with both top-left - bottom-right and bottom-left - top-right diagonals.

Return type

np.ndarray

argos.utility.reconnect(signal, newhandler=None, oldhandler=None)

Disconnect PyQt signal from oldhandler and connect to newhandler

argos.utility.rect2points(rect: numpy.ndarray) → numpy.ndarray

Convert topleft, width, height format rectangle into four anti-clockwise vertices

argos.utility.rect_intersection(ra: numpy.ndarray, rb: numpy.ndarray) → numpy.ndarray

Find if two axis-aligned rectangles intersect.

This runs almost 50 times faster than Polygon intersection in shapely. and ~5 times faster than cv2.intersectConvexConvex.

Parameters

• ra (np.ndarray) –

• rb (np.ndarray) – Rectangles specified as (x, y, w, h) where (x, y) is the coordinate of the lower left corner, w and h are width and height.

Returns

(x, y, dx, dy) specifying the overlap rectangle. If there is no overlap, all entries are 0.

Return type

np.ndarray

argos.utility.rect_ios(ra: numpy.ndarray, rb: numpy.ndarray) → float

Compute intersection over area of smaller of two axis-aligned rectangles.

This is the ratio of the area of intersection to the area of the smaller of the two rectangles.

Parameters

• ra (np.ndarray) –

• rb (np.ndarray) – Axis aligned rectangles specified as (x, y, w, h) where (x, y) is the position of the lower left corner, w and h are width and height.

Returns

The Intersection over area of the smaller of two rectangles.

Return type

float

argos.utility.rect_iou(ra: numpy.ndarray, rb: numpy.ndarray) → float

Compute Intersection over Union of two axis-aligned rectangles.

This is the ratio of the are of intersection to the area of the union of the two rectangles.

Parameters

• ra (np.ndarray) –

• rb (np.ndarray) – Axis aligned rectangles specified as (x, y, w, h) where (x, y) is the position of the lower left corner, w and h are width and height.

Returns

The Intersection over Union of two rectangles.

Return type

float

argos.utility.setStyleSheet(stylesheetname)

Set stylesheet from the _stylesheets resource (from https://github.com/Alexhuszagh/BreezeStyleSheets).

NOT USED BECAUSE THIS IS UNUSABLE!

argos.utility.tlwh2xyrh(rect)

Convert top-left, width, height into center, aspect ratio, height

argos.utility.to_qpolygon(points, scale=1.0)

Convert a sequence of (x, y) points into a qg.QPolygonF.

argos.utility.xyrh2tlwh(rect: numpy.ndarray) → numpy.ndarray

Convert centre, aspect ratio, height into top-left, width, height format

Common utility functions and constants for capturing video

argos.caputil.get_camera_fps(devid, width, height, fps=30, nframes=120)

Find the approximate frame rate of camera.

Try to capture some frames at specified size and frame rate, and calculate the actual frame rate from the time taken.

Parameters

• devid (int) – Camera number.

• width (int) – Suggested frame width.

• height (int) – Suggested frame height.

• fps (float) – Suggested frame rate (frames per second).

• nframes (int) – Number of frames to record for estimating average frame rate.

Returns

• fps (float) – Estimated frame rate of the camera.

• width (int) – Actual frame width.

• height (int) – Actual frame height.

argos.caputil.get_roi(input_, width, height)

Select region of interest (ROI) to actually save in video.

This function opens video or camera specified by input_ and shows the first frame for ROI selection. The user can left click and drag the mouse to select a rectangular area. Pressing Enter accepts the ROI selection, C cancels it. This function keeps updating the frame until a valid ROI is accepted.

Parameters

• input (str) – input video or device (a numeric string is considered camera number, a file path as a video file.

• width (int) – suggested frame width for frame capture from camera

• height (int) – suggested frame height for frame capture from camera

Returns

(x, y, w, h, width, height) where the first four specify ROI, the last two actual frame width and height.

Return type

tuple

argos.caputil.get_video_fps(fname)

Retrive the frame rate of video in file fname

Parameters

fname (str) – File path.

Returns

• fps (float) – Frame rate of video (frames per second).

• width (int) – Actual frame width.

• height (int) – Actual frame height.

class argos.vreader.VideoReader(path: str, width=- 1, height=- 1, fps=30, waitCond: Optional[threading.Event] = None)

Utility to read the video frames.

Needs a separate thread to avoid blocking the main UI

read()

Read a single frame

class argos.writer.CSVWriter(filename, mode='w')

Not used any more. Using HDF5 is much faster

reset()

Overwrite current files, going back to start

class argos.writer.DataHandler(filename, mode='w')

class argos.writer.HDFWriter(filename, mode='w')

reset()

Overwrite current files, going back to start.

Segmentation functions

argos.segment.extract_valid(points_list, pmin, pmax, wmin, wmax, hmin, hmax, roi=None)

Filter valid objects based on size limits.

The length of the smaller side of the minimum bounding rotated-rectangle is considered width and the larger as height.

Parameters

• points_list (list[np.ndarray]) – List of coordinate arrays for pixels in each segmented object pixels.

• pmin (int) – Minimum number of pixels.

• pmax (int) – Maximum number of pixels.

• wmin (int) – Minimum width of minimum bounding rotated rectangle.

• wmax (int) – Maximum width of minimum bounding rotated rectangle.

• hmin (int) – Minimum height/length of minimum bounding rotated rectangle.

• hmax (int) – Maximum height/length of minimum bounding rotated rectangle.

Returns

Coordinate arrays of objects that are between pmin and pmax pixels, wmin and wmax width, and hmin and hmax height where The length of the smaller side of the minimum bounding rotated-rectangle is considered width and the larger as height.

Return type

list

argos.segment.get_bounding_poly(points_list: List[numpy.ndarray], style: argos.constants.OutlineStyle) → List[numpy.ndarray]

This returns a list of bounding-polygons of the list of points in points_list.

Parameters

• points_list (list) – List of point arrays masking each object.

• style (argos.utility.OutlineStyle) –

Returns

• list[np.ndarray] – If style is OutlineStyle.fill - the same list of points without doing anything. If OutlineStyle.contour - the list of points representing the contour of each entry in points_list. If OutlineStyle.minrect - the list of vertices of the minimum-rotated rectangles bounding each entry in points_list. If OutlineStyle.bbox - the list of vertices of the axis-aligned rectangles bounding each entry in points_list.

• This does not strictly extract bounding points, as when style is

• OutlineStyle.filled, it just returns the same set of points. Any client

• using a uniform policy of drawing a polygon with the returned points will be

• essentially filling it up.

• I had to make a binary image with the specified points set to 1 because

• that’s what cv2.findContours takes.

argos.segment.segment_by_contour_bbox(binary_img: numpy.ndarray) → List[numpy.ndarray]

Segment binary image by finding contours of contiguous nonzero pixels. This is faster than filling the contours and then matching point labels. The bounding box is computed later on, and that is the only data saved at the end. [Although original design was to extract all pixel positions on the object, that turned out to be inefficient and unnecessary for current focus].

Parameters

binary_img (numpy.ndarray) – binary input image (obtained by thresholding grayscale image).

Returns

List of coordinate arrays where the n-th entry is the array of points defining the contour of the n-th segmented object.

Return type

list

argos.segment.segment_by_contours(binary_img: numpy.ndarray) → List[numpy.ndarray]

Segment binary image by finding contours of contiguous nonzero pixels and then filling those contours with an integer color value.

Although, this is also part of the watershed algorithm, for small objects that are clearly separable from the background and each other, this works equally well.

Parameters

binary_img (numpy.ndarray) – binary input image (obtained by thresholding grayscale image).

Returns

List of coordinate arrays where the n-th entry is the array of positions of the pixels belonging to the n-th segmented object.

Return type

list

argos.segment.segment_by_dbscan(binary_img: numpy.ndarray, eps: float = 5.0, min_samples: int = 10) → List[numpy.ndarray]

Use DBSCAN clustering to segment binary image.

Parameters

• binary_img (np.ndarray) – binary image, a 2D array containing 0s and 1s (obtaind by thresholding original image converted to grayscale).

• eps (float) – the epsilon parameter of DBSCAN.

• min_samples (int) – minimum number of pixels each cluster (object) must contain in order to be considered a valid object.

Returns

List of coordinate arrays where the n-th entry is the array of positions of the pixels belonging to the n-th segmented object.

Return type

list

argos.segment.segment_by_watershed(binary_img: numpy.ndarray, img: numpy.ndarray, dist_thresh: float = 3.0) → Tuple[numpy.ndarray, List[numpy.ndarray]]

Segment image using watershed algorithm.

Parameters

• binary_img (np.ndarray) – Binary image derived from img with nonzero pixel blobs for objects. This is usually produced after converting the img to grayscale and then thresholding.

• img (np.ndarray) – Original image to be segmented.

• dist_thresh (float, optional) – Threshold for distance of pixels from boundary to consider them core points. If it is < 1.0, it is interpreted as fraction of the maximum of the distances.

Returns

points_list – List of arrays containing positions of the pixels in each object.

Return type

list[np.ndarray]

Notes

This code is derivative of this OpenCV tutorial:

https://docs.opencv.org/master/d3/db4/tutorial_py_watershed.html

and falls under the same licensing.

Interface to YOLACT for segmentation

YOLACT was originally developed and published by Bolya et al 1.

References

1

Bolya, D., Zhou, C., Xiao, F. & Lee, Y. J. YOLACT: Real-time Instance Segmentation. (2019).

exception argos.yolactwidget.YolactException(*args, **kwargs)

class argos.yolactwidget.YolactWidget(*args, **kwargs)

process(image: numpy.ndarray, pos: int) → None

If network has not been instantiated, ask the user to provide configuration and weights.

pos - for debugging - should be frame no.

class argos.yolactwidget.YolactWorker

process(image: numpy.ndarray, pos: int)

:returns (classes, scores, boxes)

where boxes is an array of bounding boxes of detected objects in (xleft, ytop, width, height) format.

classes is the class ids of the corresponding objects.

scores are the computed class scores corresponding to the detected objects. Roughly high score indicates strong belief that the object belongs to the identified class.

setOverlapThresh(value)

Merge objects if their bboxes overlap more than this.

Wrapper to use CSRT algorithm from OpenCV to track multiple objects

class argos.csrtracker.CSRMultiTracker(*args, **kwargs)

find_nonoverlapping(bboxes)

Remove entries which are within maxDist distance from another bbox considering them to be the same object detected twice.

track(frame: numpy.ndarray, bboxes: numpy.ndarray, pos: int) → None

Track objects in frame, possibly comparing them to bounding boxes in bboxes

This uses a hybrid of CSRT with Hungarian algorithm.

Parameters

• frame (np.ndarray) – Image in which objects are to be tracked.

• bboxes (np.ndarray) – Array of bounding boxes from segmentation. In case there are no existing trackers (first call to this function), then initialize one tracker for each bounding box.

• pos (int) – Frame position in video. Not used directly, but emitted with the results via sigTracked so downstream methods can relate to the frame.

class argos.csrtracker.CSRTWidget(*args, **kwargs)

Wrapper widget providing access to CSRT parameters and to run it in a separate thread.

Since this tracker requires both the image and the bounding boxes of the segmented objects, but upstream classes are designed to send them separately (frame from VideoWidget and via sigFrame and bboxes from any of the segmentation widgets via sigProcessed signal, we need to synchronize them. This is done by two slots, setFrame and setBboxes which, track the frame position for the image and that for the bboxes and emits the data via sigTrack when the two match. The position variables are reset after that.

setBboxes(bboxes: numpy.ndarray, pos: int) → None

Slot to store bounding boxes and frame position, and signal the tracker if the image for the same frame is available

setFrame(frame: numpy.ndarray, pos: int) → None

Slot to store video frame and frame position, and signal the tracker if the bboxes for the same frame are available

class argos.csrtracker.CSRTracker(frame: numpy.ndarray, bbox: numpy.ndarray, tid: int)

Wrapper around OpenCV CSRT to maintain age information and reinitialization against segmentation data.

reinit(frame: numpy.ndarray, bbox: numpy.ndarray) → None

Reinitialize tracker to specified bounding box in frame