3.3.4 NoSQL Data Sets

Abbreviations, used in manual:

QG - Quality Gates

In QG module in the left side menu  open "My Projects" section, open required Project, open "Library" section, open "Data Sets" section. Data Sets grid for chosen Project will be shown. In order to create new Data Set click "Add Data Set" button:

Data Entity - MongoDB example:

Entity – displays a list of collections from the Data Source path, select the collection that will be used for the Data Set.

Friendly name - an alias that will be used instead of the original name in the "Selected columns" area in the column "Table".

Advanced - allows to use a Variable instead of collection name. Should be in format @variable_name;. For example:

allow disk use check box -  when type "aggregate" is chosen, mark this check box in order to allow writing temporary files on disk when a pipeline stage exceeds the 100 megabyte limit, ensuring that memory-intensive operations complete successfully. (available from quilliup version 2.7.1).

Set batch size (available for mongoDB only) - the user can set a batch size (available from quilliup version 2.7.1).

Query Type – you can choose between two query types: find and aggregate. The query content must always be enclosed in square brackets [ ], and the system will automatically prepend the selected function.
 

1. find - the find query type retrieves all documents in a collection that match the provided query conditions. You can also specify which fields to include or exclude:

 

For example:

• Filter by a field value:
  [{"car_make": "Mazda"}] - returns documents where "car_make" is "Mazda".
• Filter using multiple conditions ($and operator):
  [{"$and": [{"car_make": "Mazda"}, {"model": 1994}]}] - returns documents where "car_make" is "Mazda" and "model" is 1994.
• Filter using comparison operators ($gt - greater than):
  [{"model": { "$gt": 1994 }}] - returns documents where the "model" year is greater than 1994.
• Exclude a field from the result:
  [{}, {"price": 0}] - returns all documents but hides the "price" field.
• Filter and exclude fields together:
  [{"id": 2}, {"price": 0}] - returns the document where "id" is 2 but excludes the "price" field.
• Use logical operators ($or) with field exclusion:
  [{"$or": [{"id": 2}, {"car_make": "Mazda"}]}, {"price": 0, "model": 0}] - returns documents where "id" is 2 OR "car_make" is "Mazda", while excluding "price" and "model" fields.
   

For more advanced filtering, refer to MongoDB’s find method documentation.

2. aggregate - uses a multi-stage pipeline to process and transform documents. It allows operations such as sorting, grouping, and counting.:
 

For example:

• Sort by "id" in descending order: [{"$sort":{"id":-1}}]
   

• Group by "car_make" and count occurrences: [{"$group": {"_id": "$car_make", "count": {"$sum": 1}}}]
   

   
• Filter and project specific fields: [{"$match": {"car_make": "Mazda"}}, {"$project": {"model": 1, "_id": 0}}]
  Returns only the "model" field for documents where "car_make" is "Mazda", hiding the "_id" field.

For more aggregation options, refer to MongoDB’s aggregate method documentation.

Refresh columns button – will display updated columns list in the Selected Columns area from the right.

Validate Query button - check if inserted query is valid, for example:


 

Selected Columns area:

• Mark Checkbox from the left side to select the column(s) to be participated in the Data Set.
• Friendly - an alias that will be used instead of the original column name can be chosen.
• Data Type – the data type can be selected from the list.
• Format - depending on chosen value in "Data Type" column, available format options will appear. This is a template for displaying data that allows  to select and specify their preferred display format: