Yio Remote Community

Adding unified logging

"log": "debug" for the integration

@marton It is the wrong place her, but I found 2 things:

  • Features now cannot contain blanks. So in the config.json for “remote” “POWER TOGGLE” and “CHANNEL UP” must be changed to “POWER_TOGGLE” and “CHANNEL_UP”.

  • In the previous version a new config.json was copied automatically to the “appPath”. Now it seems that it is not done.

@marton

Until web configurator supports control of logging you can use my simple simulator app to play with logging. “QUEUE” must be enabled to display log in the simulator app.

Thanks @ChristianRiedl!
The config.json is not updated yet on github I can see. It is written with underscore here.

@ChristianRiedl just to be sure:
when a log entry is made, will teh timestamp be in utc, or in local time? and if local time, can it be changed to utc (or utc added in () ?) because, when there are communication errors with external sources and this is placed into the logfiles and a user submit such a report, it makes it easier to determine the source of failure whan one time is used instead of checking which timezone the remote currently is using.

if you need further informations or did have other questions, let me know.

@Carp3-noctem Ok, I will change it to UTC.

1 Like

@marton @zehnm

I pushed feature/apilogger branch and hope that finds grace. :grinning: :grinning:

Sorry, I can not help it, I have again dealt with the logger functionality. Basically:

  • log files can be an intermediate solution it can be replaced by journald in future. I think it is a good compromise now to create daily log files. They are purged when new log files are created.
  • we should have a real time log for example in web-configurator via yio api (currently implemented with queue log target).
  • we should have the possibility to control per category log levels online. Using Qt standard category logging using Q_LOGGING_CATEGORY(CLASS_LC, “categoryname”) creates a small problem as CLASS_LC is a const reference (see https://bugreports.qt.io/browse/QTBUG-59853 ). The standard Qt way (filterRules, init files) is not appropriate for our purpose I think. I think using a const_cast as work arround is OK.

My changes in logger. after a lot of experimenting and testing :

  • Adding a Mutex for log file writing to make it threadsafe because I noticed sporadic errors, they disappeared after this change.
  • Adding a log file prefix. Log file name is now <prefix>YYYY-MM-DD.log
  • Removing m_logLeveMask (not necessary), m_logLevel is the default loglevel, for category logs
  • constructor : change purgeHours to purgeDays, add log file prefix, remove purgeFiles from constructor. It is done in writeFile.
  • destructor : was incomplete, added some more cleanup
  • processMessage : remove oring with m_logLevelMask because it is allready done by category initisalization
  • writeFile : Make threadsafe with QMutex, create QTextStream only when creating a new file, call purgeFiles when a new log file is created, it is a good place for calling it, I think. So we have now “rolling files”. Alternatively purgeFiles could be called from some cyclic task.
  • purgeFiles : If parameter purgeDays is left, inital value is used. Considers log file prefix. Avoids deleting current log file.
  • getFileCount : Consider prefix

Explanations about defineLogCategory :

Used to define a loglevel for a category other than default log level and to optimize message logging modifying the QLoggingCategory object, why :

processMessage contains a per category log level filter (QMsgType). So filtering works additionally to the QLoggingCategory filtering. But especially in integrations qCDebug will typically log the whole communication which can create a big load only for creation of the log messages. To avoid it you can use a construct like

if (m_logCategory.isDebugEnabled()) {
    qCDebug(m_logCategory) << "onReceived " << QString(message.topic());
}

This is the reason why in Integrations::createInstance defineLogCategory should be called. Yes Logger is something generic, but PluginInterface too, and I see no other possibility to implement this feature.

if (interface) {
    connect(interface, &PluginInterface::createDone, this, &Integrations::onCreateDone);

    // Define logger category for plugin, allow logger to call plugin->setLogEnabled
    Logger* logger = Logger::getInstance();
    QString category = map.value("type").toString();
    if (logger != nullptr && category.count() > 0) {                 // category name is plugin type !!
        int level = logger->toMsgType(map.value("log").toString());  // initial log level
        logger->defineLogCategory(category, level, nullptr, interface);
    }

    interface->create(map, entities, notifications, api, config);
}

When you want to optimize logging inside remote-software you must use the const_cast or you avoid Q_LOGGING_CATEGORY like in the integrations :

static Q_LOGGING_CATEGORY(CLASS_LC, "bluetooth");

Logger::getInstance()->defineLogCategory("bluetooth", QtMsgType::QtDebugMsg, const_cast<QLoggingCategory*>(&CLASS_LC()));

Changes in config-schema.json :

  • I replaced purgeHours by purgeDays

Changes in integrations.cpp :

  • Adding defineLogCategory in Integrations::createInstance
  • Removing connect from Integrations::onCreateDone

Changes in ““entities.cpp”” :

  • Adding connect in Entities::load

Changes in main.cpp :

  • I replaced purgeHours by purgeDays

Changes in yioapi.*

  • I added apiLoggerControl

I’m on vacation right now and will have a look next week.
It would have been better to discuss logger enhancements in the open GitHub issue before making all these changes though. My idea is to let Qt & systemd handle everything once Buildroot 2020.02 is working with the updated Qt version.

I created a feature branch because current version is not working for my test environment (missing yioapi log support) and feature branch allows me to switch easily between different development environments. I added my ideas and requirements in the GitHub issue. Lets discuss it next week.

Please see my reply on GitHub.

If anyone else is following this topic, then please comment on GitHub or here on how you use or wish to use logging. Thanks!