Other systems
NDJSON import
There exists a generic data exchange format to connect any system with Makaira. The data format is based on the NDJSON format. The format holds one record in one line which makes parsing easy and effective.
{"id": "1", "type": "product", "more": "values and so on"}
{"id": "2", "type": "variant", "more": "values and so on"}
{"id": "3", "type": "manufacturer", "more": "values and so on"}
Note
For better clarity, the documents are displayed in multiple lines in the following documentation. In the final data feed each document must be delivered on separate lines without line breaks in the document itself.
General types
General types are use in every document type
Type | Description |
---|---|
timestamp | Timestamp at the time of export from the store (format: YYYY-MM-DD HH:MM:ss) |
groups | A set of data that is overridden based on the constraint "group" in the API - e.g. useful for discount groups. But also for target group-based targeting."groups": {"DISCOUNT80": {"price": 2.00, "price_no_vat": 1.68}, "DISCOUNT800": {"price": 20.00, "price_no_vat": 1.68}} |
Type product
{
"id": "10013859950",
"type": "product",
"parent": "",
"shop": "4",
"ean": "48690A",
"isVariant": false,
"active": true,
"stock": 0,
"onstock": true,
"picture_url_main": "https://images.makaira.io/out/pictures/generated/250_250/pboxx-pixelboxx-2652106/GRAPH%27IT+%22Brush+%26+Extra+fine%22+Marker+Komplett-Set%2C+Display+mit+96+Markern.jpg",
"title": "GRAPH'IT \"Brush & Extra fine\" Marker Komplett-Set, Display mit 96 Markern",
"shortdesc": "Das GRAPH'IT \"Brush & Extra fine\" enth\u00e4lt 96 Marker mit Qualit\u00e4tstinte auf Alkoholbasis und zwei unterschiedlichen Spitzen. Ideal geeignet f\u00fcr unterschiedlichste Mal- und Zeichentechniken.",
"longdesc": "Das Komplett-Set enthält alle 96 Farbtöne der GRAPH'IT \"Brush & Extra fine\" Marker. Das transparente Display ermöglicht Ihnen jederzeit gute Übersicht und einen schnellen Zugriff zu Ihren Markern.Der GRAPH'IT \"Brush & Extra fine\" Marker verfügt über zwei unterschiedliche Spitzen. Eine lange, flexible Spitze mit synthetischen Pinselfasern, die eine optimale Strömung der Tinte gewährt. Und eine extrafeine 0,5 mm-Spitze mit Metallfassung, die sich insbesondere für Illustrationen eignet.GRAPH'IT \"Brush & Extra fine\" Marker \u2013 Die Produkt-Eigenschaften im ÜberblickKorpus aus weißem Kunststoff mit Dreikantform für hervorragende Haptik und Mal-KomfortQualitätstinte auf AlkoholbasisLeicht zu öffnender Deckel mit Farbcode zur einfachen IdentifizierungDer GRAPH'IT Marker ist ideal geeignet für:ComicsArchitekturDesignLandschaftsmalereiIllustrationuvm.",
"price": 220.50,
"soldamount": 0,
"searchable": true,
"searchkeys": "GRAPH'IT \"Brush & Extra fine\", Twin Tip, Marker, Pinselmarker, Pinselstift, Stift, Doppelmarker, Kalligrafie, Twin Marker, 2 Spitzen Maker, Comic, Architektur, Design, Illustration, GRAPH'IT Marker, Set, Komplett-Set, Brushpen, Brush Pen, Sortiment",
"meta_keywords": "GRAPH'IT \"Brush & Extra fine\", Twin Tip, Marker, Pinselmarker, Pinselstift, Stift, Doppelmarker, Kalligrafie, Twin Marker, 2 Spitzen Maker, Comic, Architektur, Design, Illustration, GRAPH'IT Marker, Set, Komplett-Set, Brushpen, Brush Pen, Sortiment",
"meta_description": "Das GRAPH'IT \"Brush & Extra fine\" Marker Komplett-Set finden Sie in unserem Onlineshop. \u2713 Marker mit Qualit\u00e4tstinte auf Alkoholbasis und zwei unterschiedlichen Spitzen. \u2713 Ideal f\u00fcr unterschiedlichste Mal- und Zeichentechniken. \u2713 \u00dcberzeugen Sie sich selbst! \u2794 Jetzt g\u00fcnstig online kaufen.",
"manufacturerid": "f58f91bac483a6ea628e9d2be7a4b1e1",
"manufacturer_title": "pebeo",
"url": "/GRAPH-IT-Brush-Extra-fine-Marker-Komplett-Set-Display-mit-96-Markern.html",
"maincategory": "9fb3ea945468f7c61b471c7401e7bad4",
"maincategoryurl": "/Zeichnen-Grafik-Design/",
"groups": {
"DISCOUNT80": {
"price": 2.00,
"price_no_vat": 1.68
},
"DISCOUNT800": {
"price": 20.00,
"price_no_vat": 1.68
}
},
"category": [
{
"catid": "cb38ac47b6005b97cd1bbed61c70fb18",
"shopid": [
"4"
],
"path": "shop\/unser-angebot\/grafik-design\/stifte-marker\/copic-layoutmarker\/",
"title": "Layoutmarker",
"pos": 9 # position of product in category
}
],
"attributes": [
{
"d6238d67c4448e37c6bff847b097826e": [
"Nur Neuheiten anzeigen"
],
"5f65a4212ad959feb266af566d8f3b8f": [
"Grafik + Design"
],
"852264d68055531b4018323a7d2ba706": [
"Einsteiger",
"Schulen",
"Studenten",
"Hobbyk\u00fcnstler"
],
"a33c9d88a89257f81eef0f4a001cc365": [
"Marker + Fineliner"
],
"4c9369a9a485d7ce007f58fa695e5812": [
"Tinte auf Alkoholbasis"
],
"d8008f87168439ac398f6927cdbd812d": [
"Set"
],
"6d9b20c89a0ffb28b56bb07ff5bb39ba": [
"Pinselspitze",
"metallgefasste Spitze"
],
"76c7e398b67f441fdc9220e11b419945": [
"0,5 mm"
]
}
],
"attributeStr": [
{
"id": "d6238d67c4448e37c6bff847b097826e",
"title": "Neuheiten",
"value": "Nur Neuheiten anzeigen"
},
{
"id": "5f65a4212ad959feb266af566d8f3b8f",
"title": "Empfohlen f\u00fcr Technik",
"value": ["Grafik + Design", "Einsteiger", "Schulen", "Studenten", "Hobbyk\u00fcnstler"]
},
{
"id": "a33c9d88a89257f81eef0f4a001cc365",
"title": "Produktart",
"value": "Marker + Fineliner"
},
{
"id": "4c9369a9a485d7ce007f58fa695e5812",
"title": "Marker-Basis",
"value": "Tinte auf Alkoholbasis"
},
{
"id": "d8008f87168439ac398f6927cdbd812d",
"title": "Auswahl",
"value": "Set"
},
{
"id": "6d9b20c89a0ffb28b56bb07ff5bb39ba",
"title": "Form Marker (Spitze)",
"value": ["Pinselspitze", "metallgefasste Spitze"]
},
{
"id": "76c7e398b67f441fdc9220e11b419945",
"title": "Strichbreite",
"value": "0,5 mm"
}
],
"attributeInt": [],
"attributeFloat": [],
"mak_boost_norm_insert": 0.07301991452214238,
"mak_boost_norm_sold": 0.2763039747051342,
"mak_boost_norm_rating": 0,
"mak_boost_norm_revenue": 0.2664841710697083,
"mak_boost_norm_profit_margin": 0,
"timestamp": "2019-09-27 15:34:42",
}
Field definition
Field | Description |
---|---|
parent | This value must be empty for the product data type |
attributeStr[], attributeInt[], attributeFloat[] | - All attributes of the product - Unsorted list of attributes, one entry per value (or values separated with unique delimiter - Makaira separates them during import) - depending on the value type into the respective array: String (attributeStr) / Integer (attributeInt) / Float (attributeFloat) - for more information see section Attribute Handling |
attributes[] | Contains all attributes of the variants, grouped by variant (in the example only one variant exists) - id: Attribute identifier - value: List of attribute values (or values separated with unique delimiter - we separate them during import) - List of attribute values (or values separated with unique delimiter - we separate them during import) - for more information see section Attribute Handling |
mak_boost_norm | Fields with logarithmically normalized values for: Novelty (mak_boost_norm_insert) Number of sales (mak_boost_norm_sold) Rating (mak_boost_norm_rating) Sales (mak_boost_norm_revenue) Margin (mak_boost_norm_profit_margin) |
Type variant
Attention
At least one variant is always needed. If your product has no variant please create a "pseudo variant" with a pseudo ID (e.g. <parrent_id>_pseudo) and the parameter in the data isPseudo: true attached as a child to the "simple product"
{
"id": "10013860005",
"type": "variant",
"parent": "10013859950",
"shop": "4",
"ean": "48690",
"isVariant": true,
"active": true,
"sort": 4001,
"stock": 10,
"onstock": true,
"picture_url_main": "https://images.makaira.io/out/pictures/generated/250_250/pboxx-pixelboxx-2616996/GRAPH%27IT+%22Brush+%26+Extra+fine%22+Marker+Komplett-Set%2C+Display+mit+96+Markern.jpg",
"title": "GRAPH'IT \"Brush & Extra fine\" Marker Komplett-Set, Display mit 96 Markern",
"shortdesc": "Das GRAPH'IT \"Brush & Extra fine\" enth\u00e4lt 96 Marker mit Qualit\u00e4tstinte auf Alkoholbasis und zwei unterschiedlichen Spitzen. Ideal geeignet f\u00fcr unterschiedlichste Mal- und Zeichentechniken.",
"longdesc": "Das Komplett-Set enthält alle 96 Farbtöne der GRAPH'IT \"Brush & Extra fine\" Marker. Das transparente Display ermöglicht Ihnen jederzeit gute Übersicht und einen schnellen Zugriff zu Ihren Markern.Der GRAPH'IT \"Brush & Extra fine\" Marker verfügt über zwei unterschiedliche Spitzen. Eine lange, flexible Spitze mit synthetischen Pinselfasern, die eine optimale Strömung der Tinte gewährt. Und eine extrafeine 0,5 mm-Spitze mit Metallfassung, die sich insbesondere für Illustrationen eignet.GRAPH'IT \"Brush & Extra fine\" Marker \u2013 Die Produkt-Eigenschaften im ÜberblickKorpus aus weißem Kunststoff mit Dreikantform für hervorragende Haptik und Mal-KomfortQualitätstinte auf AlkoholbasisLeicht zu öffnender Deckel mit Farbcode zur einfachen IdentifizierungDer GRAPH'IT Marker ist ideal geeignet für:ComicsArchitekturDesignLandschaftsmalereiIllustrationuvm.",
"price": 220,
"soldamount": 0,
"searchable": true,
"searchkeys": "GRAPH'IT \"Brush & Extra fine\", Twin Tip, Marker, Pinselmarker, Pinselstift, Stift, Doppelmarker, Kalligrafie, Twin Marker, 2 Spitzen Maker, Comic, Architektur, Design, Illustration, GRAPH'IT Marker, Set, Komplett-Set, Brushpen, Brush Pen, Sortiment",
"meta_keywords": "GRAPH'IT \"Brush & Extra fine\", Twin Tip, Marker, Pinselmarker, Pinselstift, Stift, Doppelmarker, Kalligrafie, Twin Marker, 2 Spitzen Maker, Comic, Architektur, Design, Illustration, GRAPH'IT Marker, Set, Komplett-Set, Brushpen, Brush Pen, Sortiment",
"meta_description": "Das GRAPH'IT \"Brush & Extra fine\" Marker Komplett-Set finden Sie in unserem Onlineshop. \u2713 Marker mit Qualit\u00e4tstinte auf Alkoholbasis und zwei unterschiedlichen Spitzen. \u2713 Ideal f\u00fcr unterschiedlichste Mal- und Zeichentechniken. \u2713 \u00dcberzeugen Sie sich selbst! \u2794 Jetzt g\u00fcnstig online kaufen.",
"manufacturerid": "f58f91bac483a6ea628e9d2be7a4b1e1",
"manufacturer_title": "pebeo",
"url": "/GRAPH-IT-Brush-Extra-fine-Marker-Komplett-Set-Display-mit-96-Markern.html",
"maincategory": "9fb3ea945468f7c61b471c7401e7bad4",
"maincategoryurl": "/Zeichnen-Grafik-Design/",
"groups": {
"DISCOUNT80": {
"price": 2.00,
"price_no_vat": 1.68
},
"DISCOUNT800": {
"price": 20.00,
"price_no_vat": 1.68
}
},
"category": [
{
"catid": "cb38ac47b6005b97cd1bbed61c70fb18",
"shopid": [
"4"
],
"path": "shop/unser-angebot/grafik-design/stifte-marker/copic-layoutmarker/",
"title": "Layoutmarker"
}
],
"attributeStr": [
{
"id": "76c7e398b67f441fdc9220e11b419945",
"title": "Strichbreite",
"value": "0,5 mm"
}
],
"attributeInt": [],
"attributeFloat": [],
"timestamp": "2019-09-27 15:34:42"
}
Field definition
Field | Description |
---|---|
parent | Always the identifier of the parent product |
attributeStr[], attributeInt[], attributeFloat[] | - All attributes of the current variant - Unsorted list of attributes, one entry per value (or values separated with unique delimiter - Makaira separates them during import) - depending on the value type into the respective array: String (attributeStr) / Integer (attributeInt) / Float (attributeFloat) - for more information see section Attribute Handling |
Benefits of attributes, attributeStr, attributeInt and attributeFloat
- Variant precise filtering for filter combinations
(T-Shirt XL in black will only be displayed if this combination exists)- Title of attributes can be imported and used afterwards in the frontend
(otherwise has to be done via translations in the frontend)- Automated casting to Int/Float/String without our special suffixes for mapping data
Type manufacturer
{
"id": "f58f91bac483a6ea628e9d2be7a4b1e1",
"type": "manufacturer",
"manufacturer_title": "pebeo",
"timestamp": "2019-09-27 15:34:42",
"url": "/markenwelt/pebeo-gs/",
"active": true,
"shop": "4",
}
Type category
{
"id": "ffa7bbaff96b15da5579d99247e981bf",
"type": "category",
"active": true,
"hidden": false,
"depth": 1,
"sort": 1,
"category_title": "Tempera + Gouache Farbe",
"hierarchy": "b5664298b7fb8653356baafc2f8b5664298//baafc2f8b5664298b7fb865762918b32//ffa7bbaff96b15da5579d99247e981bf",
"subcategories": ["6c52086cb1550335611b759cdf8681bd"],
"timestamp": "2019-09-27 15:34:42",
"url": "/farben-hilfsmittel/kuenstlerfarben/tempera-gouache-farbe/",
"shop": "4",
}
Field definition
Field | Description |
---|---|
depth | - Starting from the root (depth = 1) - All categories must be provided, otherwise, no category trees can be created |
sort | - Defines soring of categories within the same depth - Set to 1 for no sorting |
subcategories | - List of all identifier of the subcategories |
hierarchy | - The category identifiers of the category path from the root category to the current category - identifier are separated by double slash "//" |
Custom document types
It is also possible to define own data types by just passing a unique type as string
which you can access by the API and display them. Additinal those documents are also searchable in the Makaira search. This can be use for example when you want to have your blog or store locations searchable.
{
"id": "9gh38eipwgwrwr442",
"type": "blogentry",
"active": true,
"title": "My first blog entry",
"raw_title": "First entry",
"searchkeys": "the,search,keys,of,my,first,blog,entry"
"timestamp": "2019-09-27 15:34:42",
"url": "/blog/my-first-blogentry",
"shop": "1"
...
<your custom attributes>
...
}
Field definition
Field | Description |
---|---|
title | Will be searched in the makaira search including wordprocessing like stemming, compound words, etc. |
raw_title | Alternative title without word processing only exact matches will hit |
searchkeys | Additional words to take into account for search |
Attribute handling
To understand the attribute handling we provide an example based on a T-Shirt. Let's assume the T-Shirt has two variants with attributes and some common attributes.
Data type | Title | Attributes | values |
---|---|---|---|
product | T-Shirt Emma | Cut (Id: "987cut"): Cut (Id: "988cut"): | "V-Neck" "O-Neck" or "A-Neck" |
variant | T-Shirt Emma S | Size (Id: "523size"): Length (Id: "029length"): Weight (Id: "736weight"): | "S" 13 2.4 |
variant | T-Shirt Emma XL | Size (Id: "523size"): Length (Id: "029length"): Weight (Id: "736weight"): | "XL" 25 3.0 |
For a valid import, the attributes need to be synchronized between the document type product and variant.
For importing these documents in Makaira we always need to have the attributes “synced” in both directions. The variants have to inherit the attributes of the product and the product has to aggregate the attributes of the variants in attributeStr
, attributeFloat
, attributeInt
(based on the type of the attribute).
Also the product (parent of variants) has to aggregate the attributes of all variants grouped by variant in attributes
– here the types are not relevant.
The resulting Makaira objects for an import have to look like below.
For better readability we focus in the example on the attribute values.
{
"type": "product", // or makaira-productgroup
"title": "T-Shirt Emma",
"attributeStr": [
{
"id": "987cut",
"title": "Cut",
"value": "V-Neck"
},
{
"id": "988cut",
"title": "Cut",
"value": ["O-Neck", "A-Neck"]
},
{
"id": "523size",
"title": "Size",
"value": ["S", "XL"]
}
],
"attributeInt": [
{
"id": "029length",
"title": "Length",
"value": [13, 25]
}
],
"attributeFloat": [
{
"id": "736weight",
"title": "Weight",
"value": [2.4, 3.0]
}
],
"attributes": [
{
"987cut": "V-Neck",
"988cut": ["O-Neck", "A-Neck"],
"523size": "S",
"029length": 13,
"736weight": 2.4
},
{
"987cut": "V-Neck",
"988cut": ["O-Neck", "A-Neck"],
"523size": "XL",
"029length": 25,
"736weight": 3.0
}
]
}
{
"type": "variant", // or makaira-product
"title": "T-Shirt Emma S",
"attributeStr": [
{
"id": "987cut",
"title": "Cut",
"value": "V-Neck"
},
{
"id": "988cut",
"title": "Cut",
"value": ["O-Neck", "A-Neck"]
},
{
"id": "523size",
"title": "Size",
"value": "S"
}
],
"attributeInt": [
{
"id": "029length",
"title": "Length",
"value": 13
}
],
"attributeFloat": [
{
"id": "736weight",
"title": "Weight",
"value": 2.4
}
]
}
{
"type": "variant", // or makaira-product
"title": "T-Shirt Emma XL",
"attributeStr": [
{
"id": "987cut",
"title": "Cut",
"value": "V-Neck"
},
{
"id": "988cut",
"title": "Cut",
"value": ["O-Neck", "A-Neck"]
},
{
"id": "523size",
"title": "Size",
"value": "XL"
}
],
"attributeInt": [
{
"id": "029length",
"title": "Length",
"value": 25
}
],
"attributeFloat": [
{
"id": "736weight",
"title": "Weight",
"value": 3.0
}
]
}
Product groups for filtering
In order to make the filter on group fields work correctly, these requirements need to be satisfied:
- This
groups
field has to be exists in the product and all of it's variants.- All groups should have the same fields (
price
andprice_no_vat
in the example).
The principle data structure will look like this:
{
// Other fields
"groups": {
"DISCOUNT80": {
"price": 2.00,
"price_no_vat": 1.68
},
"DISCOUNT800": {
"price": 20.00,
"price_no_vat": 1.68
}
}
}
DISCOUNT80
and DISCOUNT800
are group names
Revision based import
If you want to import your product data in a continuous and incremental manner over time you can provide us with a public endpoint in your infrastructure where we can pull data from. This data is polled by our importer process. For this, you need to fulfill a specific API Specification described below.
The data are pulled on a revision base - so your system has to somehow keep track of the changes in your database so that you can always answer the question in your system "what has been changed lately". For more information on this check section Fundamentals -> Revision
Fundamentals
Revision
- A revision describes a version of a document.
- Revision numbers are integers.
- No revision number should be used twice.
- Each change (create, update, delete) on a document (products, variants, categories, etc.) in your system should create a new revision number.
Poll Principle
- We are pulling the data out of your system by sending you a specific POST request to a given endpoint
- Your endpoint has to be public, but can be secured by an extra BasicAuth.
- Your endpoint has to validate our call by checking the signature (see next section "HMAC Signature")
- We are always pulling a batch of data starting from our maximum known revision number in Makaira.
- If your system doesn't send back data (maximum revision number reached) or throws an error, we are reducing the polling speed incremental from 1 second (fastest poll) down to 60 seconds.
HMAC Signature
- The endpoint can be secured by a BasicAuth but it must at least implement the check of the HMAC Signature to hinder unauthorized data access of your systems data.
- HMAC Signature is calculated by using the customer unique Makaira Secret as a signature key and the combination
nonce: request-body
as value for the signature. - The Nonce will be sent in the header
X-Makaira-Nonce
and normally contains the current unix timestamp. - The Signature will be sent in the header
X-Makaira-Hash
from the Makaira systems.
Example Code to generate the hash in JS (e.g. in Postman)
var nonce = postman.getEnvironmentVariable("nonce");
var secret = postman.getEnvironmentVariable("secret");
var hashString = nonce + ':' + request.data;
var hash = CryptoJS.HmacSHA256(hashString, secret);
Actions
The polling endpoint in your system has to support all three different actions for the correct dataflow:
getUpdates
This is the main endpoint that we need to import the data from your systems. We are asking your system to send us back a number of items and the revision number to start from (all documents with a revision number HIGHER than the one we are sending you)
POST <your-polling-endpoint>
Header X-Makaira-Hash: <our calculated hash>
Header X-Makaira-Nonce: <our random nonce to calculate hash>
{
"action": "getUpdates",
"since": 253425, // can be -1 at minimum if we build an makaira index from scratch
"count": 100, // normally between 1 and 500, max can be configured higher if needed
"language": "de" // one of the languages you provided us in "listLanguages"
}
We are expecting the following answer from your endpoint where as the data
has to follow the structure below product data type
Response:
{
"language": "de", // the requested language again
"highLoad": false, // you can send us a true here if your systems are under heavy load and we should reduce polling speed
"count": 1, // the count of documents you are sending us
"changes": [ // the array of documents with a small header (id, type, deleted, sequence) and the "data" that will be imported
{
"id": "product_-1110", // the ID of your document
"type": "product", // the type of your document
"sequence": 8159, // the revision number of your document
"deleted": false, // can be true if document was deleted - in this case data can be an empty object.
"data": {
// full document data
}
}
// ... more objects up until our requested count
]
}
listLanguages
To set up the import process correctly our importer needs to know how many languages are supported by your shop.
POST <your-polling-endpoint>
Header X-Makaira-Hash: <our calculated hash>
Header X-Makaira-Nonce: <our random nonce to calculate hash>
{
"action": "listLanguages"
}
We are expecting an array of ISO-639-1 language codes as a response e.g.:
Response:
[
"de",
"en",
"fr",
"nl"
]
getReplicationStatus
To check what the maximum revision number is (e.g. for displaying the "not yet imported/open changes" on the dashboard) we need the following action supported by your system:
POST <your-polling-endpoint>
Header X-Makaira-Hash: <our calculated hash>
Header X-Makaira-Nonce: <our random nonce to calculate hash>
{
"action": "getReplicationStatus",
"indices": [
{
//...
"language": "en",
"lastRevision": 266, // the maximum revision that we imported until now
"openChanges": null
//...
},
{
//...
"language": "de",
"lastRevision": 156, // the maximum revision that we imported until now
"openChanges": null
//...
}
]
}
We are expecting the indices array where the openChanges key was filled with the appropriate calculation of "how many documents are left to import after lastRevision
"
Response:
{
"indices": [
{
//...
"language": "en",
"lastRevision": 266, // the maximum revision that we imported until now
"openChanges": 20 // <-- this key was filled by your system
//...
},
{
//...
"language": "de",
"lastRevision": 156, // the maximum revision that we imported until now
"openChanges": 130 // <-- this key was filled by your system
//...
}
]
}
Updated about 2 years ago