Yio Remote Community

Adding unified logging

@marton:

I started to add “unified logging” to the Logger class in my dev branch :

  • I installed a logger function messageOutput which collects all qDebug etc and QML Logger.write output to a central point (also for the integrations).
  • logLevel decides, which message will be logged
  • messages go to the QtCreator debug output if debug is set
  • messages are written to a file if a log file is specified
  • i added writeDebug, writeInfo, and writeWarning for use in QML
  • the parameters logLevel, debug and log file are specified in the Logger constructor (still compatible with current version).
  • Logger should be created in the very beginning of main.cpp
  • Logger categories are shown in the log, filtering can be added later.

That’s great @ChristianRiedl!

Would you be able to create a pull request for this to the dev branch? So we all could try it out.

Hm, I have now several commits in my branch. I am not experiencedwith git cherry picking to pull only a part of the changes. How to continue ? I think in future it is better to create a branch for every feature. But when I work on several features, how to switch between the branches ? I think I need some git training. In the past I use git only as private store, not for working in teams. Sorry

No problem. Will fix this pull request.

Creating a feature branch is the way to go. If you don’t want to deal with the command line, there’s a github desktop app, that makes it easy to switch between branches.

@ChristianRiedl I’ve merged the pull request manually. Thanks for the great work. The logger is a blast!

That’s one of Git’s strengths. I usually work on multiple features branches in the same repo. All you need is git checkout <branch-name> and git checkout - to switch back to the previous branch. Since I have been burnt a few times with visual Git clients in the past I use the command line most of the time. You can configure graphical editors and diff tools for a little bit more comfort if you’re not a command line warrior.
Another life saver is git stash -u and git stash pop to quickly move files between branches which are not yet commited.
Personally I love the git flow extension which makes working with feature branches even simpler and use it every day.
See: https://nvie.com/posts/a-successful-git-branching-model/ and https://danielkummer.github.io/git-flow-cheatsheet/
If I have a lot of things to commit, merge or cherry pick I use Visual Studio Code (just don’t enable the “convenience” features like auto synchronization or combined commit & push). My colleagues use https://www.sourcetreeapp.com/ but I have no experience with it.
Github also offers some convenient auto link feature if you reference an issue number or pull request in the commit message. E.g. auto-closing issue once the pull-request is merged into master etc. See: https://help.github.com/en/github/writing-on-github/autolinked-references-and-urls
Just ask if you have questions.

1 Like

@marton @Nklerk
I am not only a “standardizer”, i am also an “infrastructure” guy. I am thinking about ideas I implemented in my “vue-pwa” based solution. I think it would be nice to see the “live” error messages in the web configurator alternatively to the log file on the yio remote. The feature could be enabled/disabled via a yioapi command, maybe with some filter (category, level). And then the Logger will send error messages (json) via yioapi to the web configurator. I can also add some raw implementation to the vuejs web configurator if you like. What do you think ? If you have other “jobs” for me, tell it. Today I was working hard in my job on shit documentation. I need some pleasure for the evening.

1 Like

Having log output to the web configurator would be a great addition. However the vuejs version is not the one we should use just yet. Shepless from the chat made a more vue version of it with some modules we could use. I’ll try to contact him again to push his work to the github repo so we can pick up on it. You can do the YIO API side if you want to.

If it’s for pleasure you should do something you’d like to contribute :slight_smile: However we have a backlog of issues open on github, if you find something interesting there, you can do that too. I know that @zehnm is working on all the wifi related stuff now, so not those :slight_smile:

Here is a list of integrations that we are planning as well as:

Christian, Your contribution is amazing and very welcome. One part we want to get done this year is the “remote” implementation. with this i mean control over typical ir devices or IP devices through integration plugins. some work has been started to do some basic testing but nothing final.

From a technical design point I’m not sure about what path we should take and I’m really interested in your point of view.

We thought about starting with support for 1on1 remote replacement (where one device is one entity) and later build on a recipe system like the harmony or neeo have.

what i believe is that the way the UI interacts with the integration should be uniformly usable. so for the UI it shouldn’t matter if it passes commands to the dock(ir) or any other integration. I have a couple of devices in mind that will have a dynamic command set (based on setup) i’m wrapping my head around a posible solution for this as i would love to see the integration dictates the posible commands. The way we’ve started (with dock.ir) is that config.json holds the button commands statically configured. Any way there’s a lot that surrounds basic “remote” control, and i’m interested in what you believe will be the best approach.

@marton @nklerk I added new Logger!!
It is now in the dev_christianriedl !! I need some hours of testing, than I would add it to the feature branch.

Logging

The existing Logger was extended for use in QML and C++ code inside the remote
itself and also in the integration DLL’s. Qt logging system use a QtMsgType
as “severity”. Unfortunately it is not sorted as usually. The highest value has QtInfoMsg.
So I had to introduce some workarround to define a logging level to filter the loggings in this sequence :

{ QtDebugMsg, QtInfoMsg, QtWarningMsg, QtCriticalMsg, QtFatalMsg };

Features

  • Log categories based on QLoggingCategory. For every category
    • Log message counter per severity (QtMsgType).
    • Log level
  • Write functions for use from QML
  • Log messages can optionally show the source file and the line number (qDebug feature).
  • Function to set the log level per category (setCategoryLogLevel)
  • Function to register a log category and its QLoggingCategory instance (defineLogCategory).
    When log level of a category is changed the QLoggingCategory instance is updated accordingly.
  • I assume that an integration plugin has its own logging category. Integrations load function registers the plugins.
    So Logger class can contol also the integrations log. Integrations must implement the new setLogEnabled function.
  • Log targets can be individually enabled, disabled
    • debug : The Qt Creators console
    • file : A log file (created every new hour), with purge function
    • queue : A queue with defineable maximum size. It is used to send messages to the web_configurator.
      It is better not to send every message individually. The web configurator has to request a package of messages (cyclically) with a specified maximum number of messages, a log level and a list of categories as filter.

Logger class

Q_PROPERTY  (int        logLevel            READ logLevel       WRITE setLogLevel)      // default log level
Q_PROPERTY  (bool       fileEnabled         READ fileEnabled    WRITE setFileEnabled)
Q_PROPERTY  (bool       queueEnabled        READ queueEnabled   WRITE setQueueEnabled)
Q_PROPERTY  (bool       debugEnabled        READ debugEnabled   WRITE setDebugEnabled)
Q_PROPERTY  (bool       showSourcePos       READ showSourcePos  WRITE setShowSourcePos)

// write log functions intended for QML
Q_INVOKABLE void        write               (const QString& msg);
Q_INVOKABLE void        writeDebug          (const QString& msg);
Q_INVOKABLE void        writeInfo           (const QString& msg);
Q_INVOKABLE void        writeWarning        (const QString& msg);

Q_INVOKABLE int         toMsgType           (const QString& msgType);

// set category log level, if category is not existing it is created
Q_INVOKABLE void        setCategoryLogLevel (const QString& category, int level);
Q_INVOKABLE int         getCategoryLogLevel (const QString& category);

// important when a plugin is unloaded
Q_INVOKABLE void        removeCategory      (const QString& category);

// for use in QML or from YIO API
Q_INVOKABLE QJsonArray  getQueuedMessages   (int maxCount, int logLevel, const QStringList& categories);
Q_INVOKABLE QJsonObject getInformation      ();

Q_INVOKABLE int         getFileCount        ();
Q_INVOKABLE void        purgeFiles          (int purgeHours);

// path :       directory for log file, if empty no log file
// logLevel :   default log level, will be overwritten by config setting.log
// debug :      output to QtCreator DEBUG
// showSource : show qDebug ... source and line
// queueSize :  maximum Queue size
// purgeHours : purge at start
explicit Logger (const QString& path, QtMsgType logLevel = QtDebugMsg, bool debug = true, bool showSource = false, int queueSize = 100, int purgeHours = 12, QObject *parent = nullptr);

// for use from C++ to register a Logging category
// used by integrations to register plugin logging
void   defineLogCategory   (const QString& category, int level, QLoggingCategory* loggingCategory = nullptr, IntegrationInterface* plugin = nullptr);

Remote-software sample (I implemented category logging for yioapi)

YioAPI::YioAPI(QQmlApplicationEngine *engine) :
    m_log("yioapi"),
    m_engine(engine)
{
    s_instance = this;
    Logger::getInstance()->defineLogCategory(m_log.categoryName(), QtMsgType::QtDebugMsg, &m_log);
}

Integration sample

class OpenWeatherFactory : public IntegrationInterface
{
Q_OBJECT
Q_PLUGIN_METADATA(IID "YIO.IntegrationInterface" FILE "openweather.json")
Q_INTERFACES(IntegrationInterface)
public:
    explicit OpenWeatherFactory(QObject* parent = nullptr);
    virtual ~OpenWeatherFactory() override {
    }
    void        create         (const QVariantMap& config, QObject *entities, QObject *notifications, QObject* api, QObject *configObj) override;
    void        setLogEnabled  (QtMsgType msgType, bool enable) override
    {   _log.setEnabled(msgType, enable);   }
private:
    QLoggingCategory    _log;
};

The QLoggingCategory _log must be passed on to the integration instances and threads (I propose in the constructor, as reference).

Important : the log category must be the same as the plugin name !!!

Initial loglevel of an integration can be configured in config.json :

"integrations": 
{
    "smarthomemqtt":
    {
        "log": "debug",
        "data":

YIOAPI integration

All logger functions can be controlled by the YIO API (type is log).

Actions :

  • start, target : (file|debug|queue)
  • stop, target : (file|debug|queue)
  • showsource
  • hidesource
  • setloglevel category, level : (debug, info, warning, ctritoical, fatal)
  • getmessages count, level, categorylist (all when missing)

sample result (time is unix time, type is severity) :

{  "type":"log",
   "messages":
   [
      {"cat":"openweather",
       "msg":"setup",
       "src":"..\\integration.openweather\\OpenWeather.cpp:206",
       "time":"1575551902",
       "type":0
      },
      {"cat":"openweather",
       "msg":"connect",
       "src":"..\\integration.openweather\\OpenWeather.cpp:226",
       "time":"1575551905",
      "type":0
      }
   ]
}
  • getinfo

sample result :

{
  "type":"log",
  "info":
  {
    "categories": 
   [
     {   
            "category":"qt.network.ssl",
            "countCritical":0,"countDebug":0,"countFatal":0,"countInfo":0,"countWarning":3,
            "level":4
    },
    {
            "category":"openweather",
            "countCritical":0,"countDebug":3,"countFatal":0,"countInfo":0,"countWarning":0,
            "level":0
    }
  ],
  "debugEnabled":true,"fileEnabled":true,"fileCount":3, "queueEnabled":true,"showSourcePos":true
  },
}

@marton I pushed the logger changes to the feature/weather branch.

The only breaking change is in the integrations plugin class (I assumed that we use one log category per plugin, not per integration instance.

void        create         (const QVariantMap& config, QObject *entities, QObject *notifications, QObject* api, QObject *configObj) override;
void        setLogEnabled  (QtMsgType msgType, bool enable) override
{
    _log.setEnabled(msgType, enable);
}
private:
    QLoggingCategory    _log;

It would be possible to make a default implementation inside IntergrationInterface but then it is not a pure interface :

// enable log category
virtual void setLogEnabled  (QtMsgType msgType, bool enable) {};  // = 0;

Wow @ChristianRiedl! This is impressive. I think you have covered all the needs one can wish for when it comes to logging! Great work! Thank you.

Could you create a separate feature branch for this? So we can keep track of which comes from where! Thanks!

@marton it is allready in the feature/weather branch.

I can do it, but I think it is simple to merge :

logger.* yioapi.* integrations.cpp integrationinterface.h is exactly the logger stuff.

Shall I put some docs to the wiki ?

Never mind then!

It would be great if you could do that in the wiki.

@Nklerk

Features and Commands

Maybe simple devices do not deliver the information which features they support. But in many cases the integrations know which features, commands are available. I think that the integration should be able to establish the list of commands and features. Especially for media players.

For IR remotes the situation is different. When you download a IR database (for example from GlobalCache) you get a list of the available commands. When you do it by IR learning you create such a list. In the YIO we should be able to directly work with the exports from IR databases.

Strategy

I think it is easy to support thousands of IR remotes replacement from the integration perspective. It is hard for the UI. It requires a composable UI similar like neeo.
But all the IR remotes deliver no response. And when there is only a POWER TOGGLE command, you have a big problem. (I installed a power metering device at my BeoSound 9000 power supply to know if it is on or off). So people require to support hundred different specific bluetooth, TCP based protocols, no chance to implement them all.

Most of the newest TV remotes are so handy. Hard to replace with the YIO. And you don’t need YIO’s biggest strength, the display. Because you have your big TV screen.

In the smart home world it is easier, when you support homey, home-assistant, Apple Homekit, openHab, domotitz you cover 80 % of the scenarios. But still it is a lot of bone work.

I propose to first concentrate on the most important smart home hubs and second on media players like Spotify. ROON is great but I found out that their audiophile users are far away from assembling a remote kit :grinning:. And they all have really good smartphone apps, hard to compete with them.

I have been talking with a lot of friends about the need for a remote like YIO.
Most of them have no problem to have 4 IR remotes on the table and 4 different smarthome apps to control all their stuff. Only some people are excited to have all in one nice handy piece.

And most important : It is necessary to make YIO more popular. Maybe via Raspberry project site. There is also a remotecentral web page discussing about PRONTO etc. But I don’t know Martons plans.

Recipes

My first idea is to have something like an integration, working as computing module, combining states/attributes of other integrations and commands from the UI and producing new entities states/attributes or commands to some integration. This computing module should provide function blocks like logical or/and, delay. Maybe using a dataflow library will be usefull.
I am interested to implement such a computing module (not the UI). You were so familar with the neeo recipe concept, I think you know best the requirements.

Implementing a nice configuration UI needs a lot of time. But in the first step I think that people who are able to assemble a YIO kit can define such logic in JSON.

I have a simple YIO test software to simulate hardware buttons and play with new logger features. You find it on my github https://github.com/ChristianRiedl/sim-yio-hardware.