Beauty and the Law

Featured

Latest in Beauty Tech from L’Oreal

L’Oreal is perennially at the forefront of beauty tech innovation. The company’s new patents provide a peek into the future to see what cutting-edge products are on the horizon. Read on for a discussion of three of L’Oreal’s new US patents issued in September.

Stop guessing when to change the brush in your powered facial cleansing device

New U.S. Patent No. 11,116,308 describes how the brush can be linked through your phone to record the number of times you use your device, and then recommend when to change the brush. The new invention could also let you know when to reorder your serum or cleanser; or be used to determine when you need new lights for your hand-held light emitting device. Since the system keeps track of your usage, it could remind you that it is time to cleanse your skin with the power brush when there is detection of lack of use.

The ‘308 patent broadly claims a “personal care device” and a method of using the personal care device, so it may also cover smart toothbrushes. Toothbrushes connected to apps that give feedback about how well you are brushing your teeth are known, but they send you replacement brushes on a regular schedule not tied to your actual use. They are not yet sophisticated enough to count the number of uses and duration of use to calculate when a replacement brush is needed. For that reason, this invention should have application to the oral healthcare industry as well as to skincare.

Book that elegant spa getaway and leave your cosmetics bag at home

U.S. Patent No. 11,120,398 recognizes the problems associated with attempting to travel with all bottles and containers of you need for your daily grooming. You can never be quite sure when you aren’t in compliance with regulations, meeting with the unpleasant fate of having your expensive potions discarded at the security checkpoint. Another problem with packing cosmetics for travel is that consumers purchase their products in larger sizes to use them longer before having to reorder them. The solution provided by the ‘398 patent is to direct the appropriate amount of the products you use to be packaged and delivered to your destination. Shampoo, deodorant, facial cleanser, perfumes and even toilet paper can be conveniently shipped ahead for you by the travel product system.

The smart travel system can include a preference determination engine and a product determination engine. The preference determination engine decides which toiletries you would like based on retail purchase history such as analyzing your Sephora purchases. Building upon your preferences, the product determination engine can finesse product choices for your travel destination and activities. For example, choosing a foundation with higher SPF if the destination is a sunny climate or choosing an appropriate moisturizer if your plans take you to a dry climate. An excellent idea, but hiccups in delivery may prevent the mini-cosmetic bottles ordered from reaching their destination. Delivery delays; mixed up orders; the front desk misplacing your package; or even the bellhop delivering it to the wrong room could all thwart the best-laid travel product supply plans of this smart system.

Coming to the Dermatologist’s Office

The first two patents related to inventions likely to be directly used by consumers, but L’Oreal’s U.S. Patent No. 11,127,176 protects a method of predictive facial aging for use in a clinic or spa setting. The system described in the patent is to give a patient a better idea of how their skin may age, and how they may slow the predicted aging process with the appropriate skincare products. Sampling discs are attached to a patient’s skin to obtain skin cells, which are analyzed for up to four relevant protein biomarkers. Based on the concentration of the biomarkers, a spiderweb-like diagram is produced which predicts likelihood of future aging based on characteristics such as redness, dehydration, visible spots, shininess and fine lines. The graphical representation allows the patient to explore how their skin conditions may change over time. Upon input of a proposed skin care regimen, a trend visualization engine can update the predicted course of aging to estimate the success of the selected skin care regimen in slowing an aged appearance. The patent also states that some of the identified protein biomarkers are indicative of whether or not a person will respond to the use of retinol for under-eye wrinkles.

Reflecting on the new ZMirror

Icon AI has created a smart makeup mirror with numerous features including display, ring light, camera, speakers, microphone, & LED lighting that can be integrated with an artificial intelligence voice assistant. With ZMirror’s display inset, you can easily watch a YouTube makeup tutorial while you apply your makeup. The mirror can be flipped from a vertical position to a horizontal position to transform ZMirror into an elegant mood light. Voice activated AI powered features allow skin diagnosis or augmented reality makeup modes, in addition to giving you the ability to check the news and the weather. or control other nearby electronics.

It’s easy to see why Icon AI’s new smart mirror has been awarded for its sleek design. Very soon the mirror will collect another award – this time a patent. Even though US Patent Publication No. US2021/0244204 only came out on August 12, a US patent has been approved and is scheduled to issue on September 7. The patent has moved through the patent process so quickly based on a granted Korean patent for the invention, using the Patent Prosecution Highway procedure.

According to the patent, in AR makeup mode a user’s skin is diagnosed and then cosmetics and skin care products may be recommended. By collecting customer data such as what cosmetics are used, cosmetic manufacturers may incorporate the data into product development and also use the data as a means of advertising, promoting and marketing. Close two-way communication with customers through the smart device is envisaged.

The functionality and beauty of this mirror make it very tempting for purchasers. But having such a device in your bathroom does bring certain questions about privacy and security to mind. The device is voice activated, meaning it is able to listen in. It has a camera so that photos or videos can be recorded. People may be hesitant to purchase a mirror which could potentially hear, see and transmit recordings of what they are doing in the privacy of their own bathrooms.

What if you forget to deactivate this device, or the device is hacked? Given its presumed location in your bathroom where photos, videos or sounds could be picked up that if recorded/transmitted could be embarrassing, privacy and security of the device will be very important. It may be possible to rely on built in electronic privacy safeguards, but a simple physical solution like offering a soundproof and lightproof cover as an accessory for ZMirror may make consumers more comfortable with purchasing it. When you aren’t using it, if you can cover it up for a measure of peace of mind and to help ensure personal privacy.

Perhaps this device is too smart; and has been overloaded unnecessarily with functionality. AI powered skin care or makeup guidance may be more suitably and safely accessed from a device which doesn’t stay in the bathroom where there is potential for it to become an electronic Peeping Tom such as your phone (see my previous blog post). To make ZMirror more palatable to those who have privacy concerns, an option to purchase it without AI features may expand the market for it. With smart features omitted, this product could be sold very successfully simply as a stylish mood light for enhancing your home decor. Although it was created in part to help you enhance your beauty, it could be sold just as successfully as a thing of beauty itself.

The Latest Virtual Makeover Technology: Personalized Advice at What Price?

Photo by Karolina Grabowska on Pexels.com

Getting the Chanel look is easier and more accessible than ever, thanks to smartphone technology. In February, Chanel introduced Lipscanner, an app that allows you to match any color you like to a Chanel lipstick, when you scan a photo of an item in your preferred color with your phone. Once the lipstick is identified based on the color you scanned, you can virtually test the suggested lipstick with Chanel’s Try On after you upload an image of your face. The company has committed not to collect your image or information from these apps, but it is clear that as the technology continues to become more powerful, it may become tempting for less conscientious developers to harvest the wealth of information users offer their favorite apps for free, unless such efforts are thwarted by data privacy laws and ethical standards for use of biometric information such as your facial image. A few states already have instituted protection with Biometric Information Privacy Acts, and two federal legislative proposals have been introduced, but not yet passed. Senate Bill 847 in the US (Commercial Facial Recognition Privacy Act) calls for an affirmative opt in for consent to use facial images. Senate Bill 3456 (Consumer Data Privacy and Security Act of 2020) also calls for affirmative consent before biometric information can be used.

Chanel is pressing even further forward with smart technology that can do facial analysis. In March, two new patents issued to Chanel. One is for a method and system to analyze facial features for a beauty consultation to deliver personalized advice. U.S. Patent No. 10,943,093 points out that for women, facial contrasts between skin and eyes, eyebrows or mouth tends to decrease with age. So properly applied makeup that increases the contrast of features can make women appear younger. Chanel’s invention scans a facial image, determines the degree of contrast and then categorizes the face in one of three groups. Depending on which group the facial image is placed in, suggestions can be made for makeup to enhance a person’s features. For images in the first group, increasing the contrast of the eye, mouth and eyebrow is recommended; in the second group, increasing mouth and eye contrast is recommended; while in the third group, increasing eyebrow and eye contrast is recommended. The contrast can be increased by either making the brow, lip or eye makeup more pronounced, or by using a lighter foundation. The program can recommend appropriate makeup products based on the assigned group.

The second new Chanel patent protects a method for building a neural network tool to determine certain qualities of an individual by analyzing an image of their face. According to U.S. Patent No. 10,956,716, the tool may be used to automate assessment of qualitative characteristics such as health, attractiveness or self-confidence. To create the tool, the first step involves training the neural network with a database of tens of thousands of facial images where the age of the person in the photo is known. Features are extracted from the photos, and age is estimated from the extracted features. The second step involves supplying the neural network with a second much smaller database of facial images that have been rated by humans for qualitative characteristics such as an estimation of health or self-confidence. The feature extraction learned by the system from the first step of training is applied to the smaller database of rated faces to automate and quantitate the qualitative characteristic as judged by humans. An estimation of health can then readily be provided by running a facial image run through the program.

The iterative learning process or training of neural networks requires a large database of images as inputs for success. Laws are beginning to protect your personal information by requiring your consent to use your image. While what is or is not legally appropriate is being hashed out, what may or may not be ethical should also be examined. When the input is your face and other information you provide such as how old you are, and/or how you are feeling emotionally when the photo is snapped, even if you expressly agree that your image may be used, it would also make sense to compensate you for the valuable data points which you are providing. Hopefully ethical considerations will soon evolve to complement legal protection in this rapidly growing area of technological innovation.

Exclusive Access Behind the Velvet Rope Based on Beauty?

Behind the Internet's velvet rope: Social networks of the rich and ...

Mirror, mirror on the wall – who’s the fairest of them all? A new patent coming out soon covers AI technology built to answer this age-old question. The patent will cover a system for providing an assessment of facial attractiveness using machine learning. US Patent Application Publication No. 2019/0272659 titled “System for Beauty, Cosmetic and Fashion Analysis” was just approved in March. One of the inventors is a professor at Northeastern University in MA, and the founder of Giaram, acquired by Shiseido in 2017. The concept of facial symmetry as the most reliable predictor of perceived beauty is well-known, but this invention utilizes an autoencoder-based framework to extract attractiveness-aware features in order to make an assessment of facial beauty. Beauty is said to be in the eye of the beholder, and therefore subjective; however, this invention transforms a subjective assessment into an objective one. The issue fee was paid on June 9, so the patent should publish shortly.

Northeastern University’s application of facial recognition to quantify beauty is not the first attempt to harness AI to turn a qualitative opinion of beauty into a quantitative decision. An AI judged beauty contest was held in 2016 where developers were invited to submit their AI software created to do an analysis of submitted images. China’s SenseTime reportedly also has developed software which ranks facial beauty.

With the capacity of neural networks and deep learning to rank beauty, many applications are feasible. Such a program could be used to grant access or special favors only to better looking people; powering a virtual bouncer that could exclude less attractive people from venues such as parties, events and gyms. It could also be used to guarantee a venue full of more attractive people by granting discounts in admission fees, hotel stays or cost of goods and services to those that the software judges to be more beautiful.

Further considering the idea of beauty being in the eye of the beholder, another invention uses deep learning to construct and apply a highly personalized face perception model. While Northeastern’s patent relies upon a universal measure of attractiveness (facial symmetry); U.S. Patent Publication No. 2019/0080012 titled “Method and System for Providing a Highly Personalized Recommendation Engine” recognizes that the ability to define a face as either desirable or undesirable is highly personal. For each individual, the perception of beauty is generated by a combination of many tangible and intangible factors. Tangible factors can include shape and size of particular features and symmetry. Intangible factors include a person’s prior encounters, emotional experiences, media influences and peer pressure. All these tangible and intangible factors led to a highly individualized decision mechanism being subconsciously applied. Using deep learning, a program is first trained to understand what an individual finds attractive by presenting images for the person to rank. Once the neural network has analyzed the choices, a personalized face perception model is created. Then, the model is used to analyze new images and predict desirability to the person, without the individual having to personally screen and rank the images. Several of the patent application’s claims to a computer-implemented method of constructing and using a personalized face perception model for a unique individual have already been allowed, so a patent could issue to the California inventors before the end of the year.

The obvious use of such a program is for a dating app. Singles can have potential dates chosen that are most suited to them, at least in terms of physical appeal, though it may be prudent to keep the saying “beauty is only skin deep” in mind, as AI-assisted over-reliance on superficial features ignores all the other elements essential to find the right significant other.

This technology may also be useful for directed marketing. Once a person’s preferences are known, it will be possible to show them ads including the types of people that have the features they’ve indicated to be more desirable or pleasing for product purchases (skin care, clothing etc). The knowledge that the program obtains about personal taste in looks could also be used to present targeted images in political messages or requests for charitable donations; or to design a virtual assistant or chatbot that a person would enjoy interacting with. While the movement to ban or pause use of facial recognition technology for surveillance grows, development will definitely continue at least for entertainment purposes. Because engaging AI for seemingly innocuous tasks like beauty ranking has so much potential to yield valuable user data, the rate of technological advances and more refined algorithms for facial recognition can only be expected to increase, even if privacy and bias concerns prevent security- related applications for now.

Smoke and Mirrors: Digitized Beauty’s Latest Inventions

Harnessing the power of artificial intelligence, innovation for the beauty industry is accelerating at warp speed. Here’s a brief description of a few of the latest developments related to using machine learning for cosmetics from U.S. patent applications published May 28.

Computer-Assisted Makeup Application

If you’d like to have the same makeup look that Jennifer Aniston rocked at the Emmy’s, TCMS Transparent Beauty of Texas’s invention can grant your wish. Building on the conventional mirror apps that are already available for your phone, algorithms have been created to allow you to choose Jennifer’s look, or other specific styles like an eighties look or a Goth look. Your phone will direct precisely where to apply makeup to achieve that look by the program. For those who don’t feel confident enough to put the makeup on themselves, a device or applicator to automatically apply a selected look is also discussed in U.S. Patent Application Publication No. 2020/0167983. The image below is from the patent application and shows another possible use of the invention – guidance to hide blemishes. You’ll be shown where they are, and after you’ve applied a concealer, they system checks your work and lets you know if you need to apply more makeup.

The TCMS Transparent Beauty application also discusses the interesting concept of a “looks ecosystem”, which is defined as a social internet-based environment that allows users to personalize their appearances. A looks ecosystem can establish an interactive relationship among and between users, style professionals and cosmetic producers of the exchange of styles, images, products, software applications and advice. The looks ecosystem can be used for group makeovers too.

Computer-Assisted Virtual Makeup

Maybe you want to look great, but not put in the time and effort involved to actually apply your makeup. Of course, there’s an app for that, and patent protection is being sought for it. U.S. Patent Application Publication No. 2020/0169674 is related to Shiseido’s Tele Beauty app (for putting virtual makeup on your image in a video call). The publication describes an information processing apparatus that allows you to be seen as though you are wearing makeup, even if you are not, taking the next logical steps from services like Zoom’s Touch Up My Appearance feature. “Phygital” fashion shows necessitated by social distancing rules could benefit from this technology, which could replace a makeup artist’s close contact for long periods of time with a catwalk model. The details of the patent application include other novel aspects. In addition to assisting you with making a choice about what makeup look to apply; Shiseido’s information processing apparatus can potentially also choose a makeup look for you, by mining user information about you such as your mental state, or your clothing and cosmetics purchase history from your phone or available online.

Digital Makeup Removal from an Image

In addition to helping you apply makeup; or applying makeup digitally, AI can also be used to subtract makeup from a facial image. Would you like to know what that supermodel or movie star looks like IRL? Northeastern University has a patent (U.S. Patent No. 10,339,685) that describes a way to analyze an image in order to detect the presence of makeup and remove it digitally from the image. It can be used to determine what type of makeup is used in a photo – everyday makeup, fashion makeup, television makeup, theatrical makeup etc. Besides discovering the truth about what a person really looks like, this system can analyze a photo for you; and provide the information to assist you to create the same look for yourself. It could also be used to assess and improve the skills of a makeup artist. As a natural extension of the system’s capabilities, it could find use for security, in uncovering the actual face of a person to be identified if makeup is used as a disguise to defeat facial recognition.

Smoke and mirrors effects courtesy of AI can bring new convenience and ease to the enhancement of beauty, but may also further blur the distinctions between reality and artifice for video and images.

Eyebrow Mapping with AI: Applications in Beauty and Beyond

Everyone wants to achieve perfect eyebrows to complete their look. The eyebrow is an attention-getting signature element of your face. Analyzing the unique nature and expressiveness of your brows with AI can be a useful tool on many levels. In the beauty industry, software to identify and modify facial features (facial morphing) is commonly used. Virtual plastic surgery involves computer programs and generated images to show you how your face could look if modified. This service was once exclusively available in the cosmetic surgeon’s office. Then apps like FaceTouchUp and BodyPlastika were developed to allow you to preview a cosmetic procedure from your phone, without scheduling an office visit. Tools like Benefit’s Brow Try On and Maybelline’s Brow Play Studio reside on corporate websites; providing customers a service that allows them to try on different eyebrows. App versions like Sephora’s Virtual Artist were a natural extension the cosmetic giants’ web-based services.

Soon a new app may be out that focuses specifically on eyebrows. U.S. Patent Application No. 2020/0126314 describes a technique for facial landmark detection to identify brows from a selfie. Here’s an image from the patent application:

The computer program removes the identified eyebrows from the image and allows you to choose a new color and shape to apply. Then you can see how these changes would look on your face. The facial morphing analysis technique may also potentially be used to preview alterations of other facial features prior to facelift, rhinoplasty, lip augmentation, eyebrow tinting/tattooing etc. The application just published on April 23; but has already been allowed, meaning a patent could issue shortly.

The ability to identify facial landmarks like the eyebrows with AI is also useful for fields other than beauty. Since signers use not only their hands, but also their bodies and faces to communicate, AI that can accurately interpret facial expressions helps computer programmers to convert American Sign Language into text and vice versa. To translate the Bible into sign language, Avodah, Inc. of TX has created computer programs that analyze not only what movements the hands of a signer are making, but also body movements and facial expressions, as described in U.S. Patent No. 10,489,639 and U.S. Patent Publication No. 2020/0104582 (just published in April). Identification of eyebrows and capturing of their movement is an element of Avodah’s innovative method for recognizing sign language communications.

Eyebrows as a biometric identifier? A more precise system for identifying eyebrows could also aid facial recognition of people in facemasks. Hanvon (Hanwang Technology Ltd) of China is a pioneer in this arena, having created software that allows identification of a person wearing a mask from an employee database. If the eyebrow can be more precisely mapped, it would have the potential to be an identifying feature of a mask wearer. Better triangulation and ability to identify the eyebrow facial landmark could augment electronic surveillance and security. The caveat to using eyebrows for identification; however, is that they can modified in terms of shape and color.

These examples on the leading edge of technological innovation demonstrate that whether for the beauty industry or beyond, study and analysis of the eyebrows with AI is an information source worth mining.