Release 2022.R1 – CIMcloud Search Service Catalog – Performance and Relevancy

Overview

Our most significant update in this release (internally known as version 4.0) focuses on increased performance and result relevancy for catalog searches.  We achieved this by creating a new search engine, as well as adding further cache optimization techniques for product and pricing data.

We discussed this update in more detail within our latest webinar, which you can find below:

[Link to Webinar]

Release date: 11/15/2022

Preface

The performance upgrade feature is turned off by default if your site is updated to version 2022.R1 or beyond.  Before you turn this feature on, you should be aware of the algorithmic changes made to the search logic and the capabilities you have to change those settings.

We will also discuss our cache improvements to speed up search results, what data is cached with this update, and how you can refresh cached data if data changes do not show up as quickly as you expect.

First we’ll discuss known tradeoffs when this option is turned on for faster and more relevant search results.

User Experience Improvements

What are the improvements?

• Faster delivery of product data
• More relevant search results

What is the expected improvement on catalog search with this turned on?

• Through our internal testing, we have found a 25% – 45% decrease in load time for catalog pages
  • Please note: Each site carries their own specific configuration and customizations that could sway you from achieving similar levels of performance gain.
    • For example, customer-specific pricing, classic live SOLR configurations, or third party search services may not see a similar performance gain.

What does more relevant results mean?

• Customers will be able to find the products they are searching for more accurately.  You may find that the number of results returned with this option turned on is less than the option turned off; however, the results returned will be more relevant to the keywords entered for their search.

Algorithm Changes

What has changed?

• Replaced internal search logic
• Focus on high-relevancy items through user keywords, and an omission of low-relevancy items
• Sequence of items in results could be different

What are the default settings?

• Fields searched:
  • sku
  • nm
  • ds
  • search_terms
  • searchfield1
  • searchfield2
  • searchfield3
  • searchfield4
  • searchfield5
  • Parent/Child Data
    • child_skus
    • child_nms
    • child_rollup_search_terms
  • Alias Data
    • sku_alias
    • nm_alias
    • ds_alias

Caching

Caching pre-processes data temporarily to make it faster to retrieve data by the user; however, there is an increase in time to reflect data changes in the database (e.g. changing a product name or price in your ERP will take longer to display to the end-user).

What is cached for SOLR indexing?

• Product Data, including:
  • Inventory Data
  • Product Account History Data
  • Faceted Search Data
• Pricing Data (customer-specific pricing)

How often is the above data cached?

• Internal search engine crawls and caches every 60 minutes
• Up to a 60 minute delay for catalog data changes to appear in searches

How can I re-index and re-cache my catalog data?

• In the Worker Portal, head to “Search Field Boosts” and submit the page.  This will force a SOLR reindex of your data, as well as re-cache it for your users
• You can also head to “System-Wide Settings” -> “Site Cache” to force a recache of all data in your staging and/or production environment
  • Note: This is done automatically when you POST the boosts page as well as when you submit any settings pages (features, customer site settings, etc)

Tradeoffs

As we work to make improvements to our Search API, there are known tradeoffs with this option turned on that are listed below:

• Delay in data updates
  • Data changes will take longer to show up on your site’s search results and product detail pages
  • Cache optimizations allow for faster delivery of product data to customers browsing your site.  With caching turned off, performance would trend downwards based off the high-volume of data used to run search queries.
  • We discuss this further in the “Caching” section below
• If you have customizations on the search results or product detail pages, there may be associated work to include them when this option is turned on
  • For example:
    • Any fields displayed, or used, that are not listed in the below default settings section will require custom development to include
    • Modifying the algorithm for searching requires an estimate from our professional services team
• Some bundles may not be supported when Search Service is enabled
  • For example: Product Configurator

How to Turn this Feature On

CIMCloud is enabling this feature on a case by case basis currently, as it is in a BETA state.  Once it is released to all customers, it will be included in the Base implementation as a new feature under Application Settings -> Product Catalog.

How to modify your results

There are two sections to be aware of when you turn the search service on:

1. Which fields can be searched and the priority you give to those fields (i.e. Boost scores)
2. How people can sort by the results via Search Service feature
   1. This is currently set by CIMcloud employees due to the formatting of the feature.  In the future, this will be opened up to customers

Boost Multipliers

Boost multipliers, and the fields that can be searchable, are set in the Faceted Search -> Search Field Boosts page (mgr_search_field_boosts_man.asp) – [Reference A] below.

On this page you can declare the searchable fields and the priority/relevancy you give to those fields (i.e. Boost multipliers).

There are three types of search matching available, where each type can be individually disabled by database field.  For example, given a product with name “Widget B – Parent with Two Choices”:

• Whole-word indexes whole words without modification
  • Matches “widget” or “choices”
• Partial-word indexes parts of words, stripping away common suffixes like “-s” or “-ing”; SKUs are indexed for any partial match
  • Matches “choice”
• Exact-field indexes the entire field, ignoring capitalization
  • Matches “widget b – parent with two choices”

Boost multipliers affect the relevancy scores:

• A multiplier can be as low as 0 (a field score of 0 will include matches in the result but will not influence relevancy ordering)
• Multipliers cannot be blank or negative

Any changes made to this page can take around 3-5 minutes to reindex the fields for search.

Note: Please submit a ticket to extranet if there is a field you would like to be included in searches that isn’t included on this page.

How Can I Test This Update?

There are a few methods to test this update on your site before rolling it out to all of your live customers:

• In a Sandbox
  • This will not have the same performance gain as on your production site, but you can check for any errors or gaps in functionality before updating your staging/production site’s version
• For B2B Customers
  • Attaching the feature to a Customer Group
• For Retail Customers
  • Attaching the feature to a Customer Site
  • Note: You may need to contact us if you are using your free customer site already.  We can add a temporary Customer Site to test this update at no cost; however, there will be an expiration on this additional site

References

[Reference A]

 

Fixes

3.7.11

• Fixes issue with saving Customer Site Email settings.
• Updates shipping estimator to display shipping errors that come back from the Ordering Object
• Fixes issue with “Confirmation Text to show if Auto-Login is disabled” not showing when the site is set to require login.
• Adds logic to disable new Customer Sites on sandbox databases.
• Updates Abanonded Cart Datamove to select only abandoned carts on the customer’s current session.
• Updates mgr_ae_inventory_level.asp to build ref_id to match ERP config’s format
• Fixes issue where nested search groups were not saving.

3.7.10

• Fixes issue with order confirmation email not generating properly

3.7.9

• Fixes issue with favorite list nicknames that contained percent signs.
• Added DashboardActionTop hook to dashboard.asp.
• Updates order_confirmation.asp to pull OD quanties when summing, from the [order_shipments] table.
• Fixes issue when clear cache is automatically called on certain edit pages while using a local development site.
• Adds ref_id to the label on Multi-Input when selecting accounts for member groups.
• Fixes data discrepency on mgr_manage_orders.asp page.
• Fixes search timeout on addresses.asp.
• Adds an exception if two order payment profile records exist.
• Adds external link to google font list on design settings.
• Fixes Example Font formatting on design settings.
• Adds message to top of Customer Site and Group Features page when a feature is hidden because it’s set to not allow overrides (assignable = 0).
• Updates Help Tip Text on mgr_un_pw_requests_man.asp page.

3.7.8

• Adds LEFT 128 to the connection string APP name since. The property has a max length of 128 so this will prevent errors if the script name is too long.
• Updates “New Account” confirmation email to filter out the House User when displaying contact info.
• Adds vault url to logging in LIB_Saved_Payment_Methods.asp
• Adds support for “Additional Link Properties” on the dashboard.asp page for the sitefront.
• Fixes issue with Web Page find popup on Add-Edit Link page.
• Updates filter on su_account_detail.asp Recent Sales Orders so it excludes “Canceled” orders
• Adds support for Promotion messages while adding products to the cart from the payment page.
• Fixes issue when using promotions and sales uom at the same time.
• Adds SuperUserAccountsDetailsActionBeforeContactDDT to su_account_detail.asp page.
• Excludes Skin settings from the sitefront when the feature isn’t enabled.
• Updates the placement of “Tips & Tricks” button on customer site “Advanced” section.
• Updates sitefront contact_edit.asp page to not show AQT section unless GLOBAL_USE_POWER_CUSTOMERS AND accounts_query_type is Blank, null, “my-acct”, or “account”.

3.7.7

• Fixes issue with indexing URLs that contain ?quickadd=true
• Fixes issue with the successful toast notification on Application Settings Shipping Features.
• Fixes issue with removing mapped Customer Groups on the account_edit.asp page.
• Adds order_shipments.phone field to mgr_order_header_ae.asp Shipping Address Edit popup.
• Fixes issue with tag replacement on “triggered” emails
• Fixes “update” cart link positioning on mobile.

3.7.6

• Fixes favorites list count when using datasets.
• Adds Retail Sidebar Hooks
• Fixes issue where multi-site emails were not saving the l_ws_id properly
• Fixes issue with customer site settings not being applied to “triggered” emails.

3.7.5

• Adds additional parameter source to getProducts function in json_api_catalog.asp so the origin of the request can be identified and used in hooks
• Adds SsjsApiFiltergetProducts filter to json_api_catalog.asp page
• Fixes page number calculation on the following pages:
  • mgr_invoice_lines_audit.asp
  • mgr_open_invoices_audit.asp
  • mgr_sales_order_lines_audit.asp
  • mgr_open_sales_orders_audit.asp
• Fixes Data Set UI to only show “Data Rights” on the following pages:
  • mgr_product_ae.asp
  • ae_prod_cat_2008.asp
  • mgr_shipping_method_ae.asp
• Updates the Pending Orders check on sales_orders.asp to check for multiple_ship_vias to exclude “Parent” orders.
• Adds new options for labels in the interaction tracker instead of the hard-coded labels.
• Adds new action hook NewAccountEmailReplaceContent to new_account.asp page
• Updates COM_VER_PRICINGOVERRIDES and WSP_SESSION_OBJECT_VERSION to version 364
  • Addresses rounding issues
• Fixes issues with ERP Driver Database Constraints. This replaces a manual step usually done by devops or deployment.

3.7.4

• Fixes issue with bDisallowIndexing on mycimcloud.com
• Updates “Standard Domains” on Customer Site Edit page to support the new subdomain format.
• Correcting style closing tag placement on css_overrides.asp
• Fixes typo in “no search results found” message.
• Fixes date range filter on mgr_prod_inv_watch_list_man.asp
• Removes check for deprecated options in the order_confirmation.asp page related to showing Expected and Requested Ship Date.
• Updates mgr_product_ae.asp help tip for the opt X sort fields.
• Adds Gift Certificate support to view orders page and order confirmation email.
• Fixes shipping settings not being “global”.

3.7.3

• Fixes Site Urls under “Your Domains” section of the Edit Customer Site page.
• Fixes issue during migrations where existing cookies contain invalid L_WS_ID values which causes logins to fail when using Customer/Account validation on site logins.
• Updates feature mapping logic to remove a feature toggle from any group it’s mapped to when the “Allow Default Status Override” setting is set to “no”. Setting this to “no” causes the feature to be hidden from Customer Sites and Groups and if that feature is mapped to a group already, you can no longer remove it since it’s hidden.
• Updates customer_groups.asp and worker_groups.asp pages to hide the system groups (application, workerspace and customer site settings).
• Removes validation on the shipping settings > “shipper number” field.
• Fixes issue with the country drop list showing when it shouldn’t on create_login_and_account.asp page.
• Adds faceted range search hooks (matching CFV version).
• Updates the robots meta tag to also include nofollow and noimageindex

3.7.2

• Fixes html encoding issue on category window title attributes
• Updates ATC logic to apply the default warehouse from the session when no pw_id is passed from the calling page.
• Fixes issue where parent product image “flashes” when changing the selected product in droplist view.
• Updates checkout process to use the ADDRESS_BOOK_FIELDS consistently through the checkout process
• Fixes issue with parent product key not getting set when a child product is added to cart from its own detail page
• Fixes SSL issue with the email test page on local sites.
• Fixes issue with rights check for portal home links where it was only checking against the first applied right for the user
• Fixes issue with searching quantity fields on backorders.asp page.
• Fixes issue where Add to Cart button didn’t show on product_history_report.asp page.

3.7.1

• Fixes phone number not getting updated when editing shipping address from payment page
• Updates Help Tip Placement on password reset pages to be on top of the input to make the text visible on mobile devices
• Updates Edit Group page to force refresh any sessions for users that are in that group
• Fixes issue where no records display on mgr_manage_imports_exports.asp page.